Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:55.842077 20876 test_util.cc:268] Using random seed: 1297750513
I20240503 14:11:55.853608 20876 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:11:55.853696 20876 ts_itest-base.cc:113] --------------
I20240503 14:11:55.853744 20876 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:11:55.853786 20876 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:11:55.853832 20876 ts_itest-base.cc:116] --------------
2024-05-03T14:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:55Z Disabled control of system clock
I20240503 14:11:55.879673 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:39349
--webserver_interface=127.20.99.62
--webserver_port=0
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.99.62:39349
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:11:56.208345 20888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.208671 20888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.208762 20888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.216207 20888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:56.216336 20888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:56.216390 20888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.216445 20888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:56.216493 20888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:56.225755 20888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.99.62:39349
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.99.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.227118 20888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.228914 20888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.240162 20893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.240660 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.241132 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.242074 20888 server_base.cc:1034] running on GCE node
I20240503 14:11:56.242837 20888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.244490 20888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.245750 20888 hybrid_clock.cc:648] HybridClock initialized: now 1714745516245703 us; error 42 us; skew 500 ppm
I20240503 14:11:56.246120 20888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.253144 20888 webserver.cc:460] Webserver started at http://127.20.99.62:36555/ using document root <none> and password file <none>
I20240503 14:11:56.253988 20888 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.254096 20888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.254438 20888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.260696 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a68d02bc9f094766af1cf531f7df1704"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:11:56.261487 20888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a68d02bc9f094766af1cf531f7df1704"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:11:56.266702 20888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:11:56.270048 20902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.271446 20888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:56.271607 20888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a68d02bc9f094766af1cf531f7df1704"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:11:56.271827 20888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.290930 20888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.291811 20888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.292094 20888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.310029 20888 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.62:39349
I20240503 14:11:56.310045 20953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.62:39349 every 8 connection(s)
I20240503 14:11:56.311431 20888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:11:56.315120 20954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:56.320016 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 20888
I20240503 14:11:56.320255 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:11:56.324012 20954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704: Bootstrap starting.
I20240503 14:11:56.328027 20954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:56.328961 20954 log.cc:824] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:56.331418 20954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704: No bootstrap required, opened a new log
I20240503 14:11:56.338502 20954 raft_consensus.cc:348] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d02bc9f094766af1cf531f7df1704" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 39349 } }
I20240503 14:11:56.338848 20954 raft_consensus.cc:374] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:56.339016 20954 raft_consensus.cc:729] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68d02bc9f094766af1cf531f7df1704, State: Initialized, Role: FOLLOWER
I20240503 14:11:56.339771 20954 consensus_queue.cc:260] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d02bc9f094766af1cf531f7df1704" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 39349 } }
I20240503 14:11:56.340039 20954 raft_consensus.cc:388] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:56.340170 20954 raft_consensus.cc:482] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:56.340353 20954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:56.343286 20954 raft_consensus.cc:504] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d02bc9f094766af1cf531f7df1704" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 39349 } }
I20240503 14:11:56.343858 20954 leader_election.cc:302] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a68d02bc9f094766af1cf531f7df1704; no voters:
I20240503 14:11:56.344594 20954 leader_election.cc:288] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:56.344729 20959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:56.345646 20959 raft_consensus.cc:686] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [term 1 LEADER]: Becoming Leader. State: Replica: a68d02bc9f094766af1cf531f7df1704, State: Running, Role: LEADER
I20240503 14:11:56.346239 20959 consensus_queue.cc:237] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d02bc9f094766af1cf531f7df1704" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 39349 } }
I20240503 14:11:56.348829 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a68d02bc9f094766af1cf531f7df1704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d02bc9f094766af1cf531f7df1704" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 39349 } } }
I20240503 14:11:56.349149 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.350749 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a68d02bc9f094766af1cf531f7df1704. Latest consensus state: current_term: 1 leader_uuid: "a68d02bc9f094766af1cf531f7df1704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68d02bc9f094766af1cf531f7df1704" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 39349 } } }
I20240503 14:11:56.351060 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.352378 20954 sys_catalog.cc:564] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:56.358141 20963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:56.363332 20963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:56.371964 20963 catalog_manager.cc:1347] Generated new cluster ID: a7c16080cd484d808631d0f0140b9d0c
I20240503 14:11:56.372125 20963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:56.387046 20963 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:56.388540 20963 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:56.397142 20963 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a68d02bc9f094766af1cf531f7df1704: Generated new TSK 0
I20240503 14:11:56.397958 20963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:56.417086 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.1:0
--local_ip_for_outbound_sockets=127.20.99.1
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 14:11:56.742964 20979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.743306 20979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.743472 20979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.751852 20979 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:56.751979 20979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.752149 20979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.1
I20240503 14:11:56.761611 20979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.99.1
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.1
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.762971 20979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.764737 20979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.777101 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.777633 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.778113 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.778940 20979 server_base.cc:1034] running on GCE node
I20240503 14:11:56.779625 20979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.780622 20979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.781822 20979 hybrid_clock.cc:648] HybridClock initialized: now 1714745516781765 us; error 61 us; skew 500 ppm
I20240503 14:11:56.782151 20979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.784559 20979 webserver.cc:460] Webserver started at http://127.20.99.1:34719/ using document root <none> and password file <none>
I20240503 14:11:56.785456 20979 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.785615 20979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.786100 20979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.788878 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1114934d9a3544a9935d75504bc27ceb"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:11:56.789609 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1114934d9a3544a9935d75504bc27ceb"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:11:56.794695 20979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:11:56.798049 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.799419 20979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:56.799580 20979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1114934d9a3544a9935d75504bc27ceb"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:11:56.799767 20979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.814474 20979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.815338 20979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.815616 20979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.816618 20979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.818114 20979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.818209 20979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.818346 20979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.818434 20979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.850445 20979 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.1:43389
I20240503 14:11:56.850507 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.1:43389 every 8 connection(s)
I20240503 14:11:56.851833 20979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:11:56.856952 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 20979
I20240503 14:11:56.857133 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:11:56.860381 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.2:0
--local_ip_for_outbound_sockets=127.20.99.2
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:56.864886 21106 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:56.865271 21106 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.866317 21106 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:56.868741 20919 ts_manager.cc:194] Registered new tserver with Master: 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:11:56.870940 20919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.1:43213
W20240503 14:11:57.194337 21110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.194636 21110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.194737 21110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.203238 21110 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:57.203382 21110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.203550 21110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.2
I20240503 14:11:57.212702 21110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.99.2
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.2
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.214051 21110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.215852 21110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.228435 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.229000 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.229607 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.230286 21110 server_base.cc:1034] running on GCE node
I20240503 14:11:57.230894 21110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.231918 21110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.233139 21110 hybrid_clock.cc:648] HybridClock initialized: now 1714745517233049 us; error 76 us; skew 500 ppm
I20240503 14:11:57.233479 21110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.235886 21110 webserver.cc:460] Webserver started at http://127.20.99.2:41775/ using document root <none> and password file <none>
I20240503 14:11:57.236637 21110 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.236773 21110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.237130 21110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.239763 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8cc13f87df2343dcb1c50aae26230e82"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-p0gh"
I20240503 14:11:57.240494 21110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8cc13f87df2343dcb1c50aae26230e82"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-p0gh"
I20240503 14:11:57.245764 21110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240503 14:11:57.249610 21124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.251286 21110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:57.251472 21110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8cc13f87df2343dcb1c50aae26230e82"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-p0gh"
I20240503 14:11:57.251694 21110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.273411 21110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.274346 21110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.274611 21110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.275619 21110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.277107 21110 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.277195 21110 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.277269 21110 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.277315 21110 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.310726 21110 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.2:33731
I20240503 14:11:57.310806 21236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.2:33731 every 8 connection(s)
I20240503 14:11:57.312119 21110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:11:57.319953 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21110
I20240503 14:11:57.320130 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:11:57.323916 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.3:0
--local_ip_for_outbound_sockets=127.20.99.3
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:57.326965 21237 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:57.327340 21237 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.328415 21237 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:57.329969 20919 ts_manager.cc:194] Registered new tserver with Master: 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:11:57.331092 20919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.2:36225
W20240503 14:11:57.664644 21240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.664934 21240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.665068 21240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.672585 21240 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:57.672708 21240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.672833 21240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.3
I20240503 14:11:57.682046 21240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.99.3
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.3
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.683429 21240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.685202 21240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.697443 21245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.698364 21246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.698891 21248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.699616 21240 server_base.cc:1034] running on GCE node
I20240503 14:11:57.700268 21240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.701251 21240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.702481 21240 hybrid_clock.cc:648] HybridClock initialized: now 1714745517702432 us; error 43 us; skew 500 ppm
I20240503 14:11:57.702805 21240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.705384 21240 webserver.cc:460] Webserver started at http://127.20.99.3:38641/ using document root <none> and password file <none>
I20240503 14:11:57.706133 21240 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.706251 21240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.706571 21240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.709259 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-p0gh"
I20240503 14:11:57.710115 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-p0gh"
I20240503 14:11:57.720711 21240 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20240503 14:11:57.724550 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.725953 21240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20240503 14:11:57.726125 21240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-p0gh"
I20240503 14:11:57.726343 21240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.748116 21240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.749217 21240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.749508 21240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.750526 21240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.752117 21240 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.752228 21240 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.752342 21240 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.752411 21240 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.790318 21240 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.3:43159
I20240503 14:11:57.790369 21366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.3:43159 every 8 connection(s)
I20240503 14:11:57.791719 21240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:11:57.797765 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21240
I20240503 14:11:57.797963 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:11:57.801462 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.4:0
--local_ip_for_outbound_sockets=127.20.99.4
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:57.805532 21367 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:57.805845 21367 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.806685 21367 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:57.808279 20919 ts_manager.cc:194] Registered new tserver with Master: bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:11:57.809518 20919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.3:47313
I20240503 14:11:57.874919 21106 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
W20240503 14:11:58.145697 21370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.146206 21370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.146387 21370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.154188 21370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:58.154341 21370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.154503 21370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.4
I20240503 14:11:58.164031 21370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.99.4
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.4
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.165405 21370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.167125 21370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.180187 21378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.181118 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.181527 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.182514 21370 server_base.cc:1034] running on GCE node
I20240503 14:11:58.183112 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.184083 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.185405 21370 hybrid_clock.cc:648] HybridClock initialized: now 1714745518185351 us; error 35 us; skew 500 ppm
I20240503 14:11:58.185745 21370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.188208 21370 webserver.cc:460] Webserver started at http://127.20.99.4:36951/ using document root <none> and password file <none>
I20240503 14:11:58.188953 21370 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.189056 21370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.189388 21370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.192102 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "20560106dcaf405bbb288e4e3e8affa6"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:11:58.192859 21370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "20560106dcaf405bbb288e4e3e8affa6"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:11:58.198077 21370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 14:11:58.201531 21384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.202883 21370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 14:11:58.203037 21370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "20560106dcaf405bbb288e4e3e8affa6"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:11:58.203265 21370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.217298 21370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.218154 21370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.218437 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.219540 21370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.221134 21370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.221232 21370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.221352 21370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.221426 21370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.255900 21370 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.4:42461
I20240503 14:11:58.255959 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.4:42461 every 8 connection(s)
I20240503 14:11:58.257458 21370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:11:58.261045 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21370
I20240503 14:11:58.261236 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:11:58.265012 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:0
--local_ip_for_outbound_sockets=127.20.99.5
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:58.272359 21497 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:58.272698 21497 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.273594 21497 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:58.275058 20919 ts_manager.cc:194] Registered new tserver with Master: 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:11:58.276011 20919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.4:42925
I20240503 14:11:58.333806 21237 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
W20240503 14:11:58.608042 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.608366 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.608517 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.616160 21500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:58.616315 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.616489 21500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:11:58.625916 21500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.627322 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.629089 21500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.641939 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.642563 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.643425 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.644098 21500 server_base.cc:1034] running on GCE node
I20240503 14:11:58.644693 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.645637 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.646848 21500 hybrid_clock.cc:648] HybridClock initialized: now 1714745518646803 us; error 38 us; skew 500 ppm
I20240503 14:11:58.647202 21500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.649482 21500 webserver.cc:460] Webserver started at http://127.20.99.5:38637/ using document root <none> and password file <none>
I20240503 14:11:58.650286 21500 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.650391 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.650710 21500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.653632 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "eb49440048bd4d749779867fa4ae1489"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:11:58.654439 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "eb49440048bd4d749779867fa4ae1489"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:11:58.659770 21500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:11:58.663256 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.664593 21500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:11:58.664813 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "eb49440048bd4d749779867fa4ae1489"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:11:58.665035 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.700237 21500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.701145 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.701380 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.702342 21500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.703857 21500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.703943 21500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.704025 21500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.704077 21500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.735888 21500 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:40925
I20240503 14:11:58.735955 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:40925 every 8 connection(s)
I20240503 14:11:58.737197 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:11:58.744760 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21500
I20240503 14:11:58.744938 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:11:58.748368 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.6:0
--local_ip_for_outbound_sockets=127.20.99.6
--webserver_interface=127.20.99.6
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:58.752554 21627 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:58.752903 21627 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.753798 21627 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:58.755357 20919 ts_manager.cc:194] Registered new tserver with Master: eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925)
I20240503 14:11:58.756127 20919 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:37349
I20240503 14:11:58.812392 21367 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
W20240503 14:11:59.082649 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:59.082976 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:59.083122 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:59.090747 21630 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:59.090896 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:59.091065 21630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.6
I20240503 14:11:59.100580 21630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.99.6
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.6
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:59.101971 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:59.103873 21630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:59.116976 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:59.117941 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:59.118553 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:59.119673 21630 server_base.cc:1034] running on GCE node
I20240503 14:11:59.120312 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:59.121326 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:59.122531 21630 hybrid_clock.cc:648] HybridClock initialized: now 1714745519122483 us; error 42 us; skew 500 ppm
I20240503 14:11:59.122833 21630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:59.125375 21630 webserver.cc:460] Webserver started at http://127.20.99.6:33677/ using document root <none> and password file <none>
I20240503 14:11:59.126194 21630 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:59.126295 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:59.126588 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:59.129300 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "748b2b25b1b94a1b94788e8ece4ecb6d"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:11:59.130041 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "748b2b25b1b94a1b94788e8ece4ecb6d"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:11:59.135661 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 14:11:59.139892 21644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.141331 21630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:11:59.141499 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "748b2b25b1b94a1b94788e8ece4ecb6d"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:11:59.141706 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:59.159924 21630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:59.160884 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:59.161211 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:59.162191 21630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:59.164052 21630 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:59.164158 21630 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.164314 21630 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:59.164386 21630 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.198408 21630 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.6:36299
I20240503 14:11:59.198462 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.6:36299 every 8 connection(s)
I20240503 14:11:59.199855 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:11:59.210191 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21630
I20240503 14:11:59.210397 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:11:59.213367 21757 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:59.213733 21757 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:59.214337 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.7:0
--local_ip_for_outbound_sockets=127.20.99.7
--webserver_interface=127.20.99.7
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:59.214828 21757 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:59.216502 20918 ts_manager.cc:194] Registered new tserver with Master: 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299)
I20240503 14:11:59.217360 20918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.6:45437
I20240503 14:11:59.278625 21497 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
W20240503 14:11:59.557197 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:59.557499 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:59.557653 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:59.565295 21760 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:11:59.565425 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:59.565600 21760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.7
I20240503 14:11:59.575037 21760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.99.7
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.7
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:59.576529 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:59.578488 21760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:59.593706 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:59.594259 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:59.594767 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:59.595681 21760 server_base.cc:1034] running on GCE node
I20240503 14:11:59.596299 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:59.597288 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:59.598502 21760 hybrid_clock.cc:648] HybridClock initialized: now 1714745519598461 us; error 37 us; skew 500 ppm
I20240503 14:11:59.598842 21760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:59.601208 21760 webserver.cc:460] Webserver started at http://127.20.99.7:43911/ using document root <none> and password file <none>
I20240503 14:11:59.601945 21760 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:59.602046 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:59.602375 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:59.605113 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "ad5e6850bbff47fd9ff2e14de5b48223"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:11:59.605863 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "ad5e6850bbff47fd9ff2e14de5b48223"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:11:59.611135 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:11:59.614531 21774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.615927 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:59.616084 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ad5e6850bbff47fd9ff2e14de5b48223"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:11:59.616283 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:59.634652 21760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:59.635567 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:59.635831 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:59.636777 21760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:59.638363 21760 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:59.638458 21760 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.638571 21760 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:59.638645 21760 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:59.673658 21760 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.7:43693
I20240503 14:11:59.673730 21886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.7:43693 every 8 connection(s)
I20240503 14:11:59.674933 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:11:59.683604 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21760
I20240503 14:11:59.683821 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:11:59.687410 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.8:0
--local_ip_for_outbound_sockets=127.20.99.8
--webserver_interface=127.20.99.8
--webserver_port=0
--tserver_master_addrs=127.20.99.62:39349
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:11:59.689129 21887 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:11:59.689510 21887 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:59.690582 21887 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:11:59.692224 20918 ts_manager.cc:194] Registered new tserver with Master: ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693)
I20240503 14:11:59.692992 20918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.7:35907
I20240503 14:11:59.758980 21627 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
W20240503 14:12:00.035705 21890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:00.036007 21890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:00.036111 21890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:00.044548 21890 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:00.044694 21890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:00.044886 21890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.8
I20240503 14:12:00.054620 21890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.99.8
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.8
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:00.055989 21890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:00.057797 21890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:00.070489 21898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:00.071224 21896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:00.071938 21895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:00.072901 21890 server_base.cc:1034] running on GCE node
I20240503 14:12:00.074107 21890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:00.075081 21890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:00.076303 21890 hybrid_clock.cc:648] HybridClock initialized: now 1714745520076259 us; error 32 us; skew 500 ppm
I20240503 14:12:00.076655 21890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:00.079011 21890 webserver.cc:460] Webserver started at http://127.20.99.8:41365/ using document root <none> and password file <none>
I20240503 14:12:00.079810 21890 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:00.079941 21890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:00.080258 21890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:00.082856 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "bb8d27ca78834bacb9ef4a67a829b494"
format_stamp: "Formatted at 2024-05-03 14:12:00 on dist-test-slave-p0gh"
I20240503 14:12:00.083660 21890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "bb8d27ca78834bacb9ef4a67a829b494"
format_stamp: "Formatted at 2024-05-03 14:12:00 on dist-test-slave-p0gh"
I20240503 14:12:00.088999 21890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:00.092492 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:00.094002 21890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:00.094175 21890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bb8d27ca78834bacb9ef4a67a829b494"
format_stamp: "Formatted at 2024-05-03 14:12:00 on dist-test-slave-p0gh"
I20240503 14:12:00.094386 21890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:00.113253 21890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:00.114195 21890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:00.114473 21890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:00.115523 21890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:00.117141 21890 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:00.117246 21890 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:00.117352 21890 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:00.117420 21890 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:00.150763 21890 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.8:40487
I20240503 14:12:00.150836 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.8:40487 every 8 connection(s)
I20240503 14:12:00.152118 21890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:12:00.157225 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 21890
I20240503 14:12:00.157438 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:12:00.166059 22017 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:12:00.166363 22017 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:00.167112 22017 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:12:00.168496 20918 ts_manager.cc:194] Registered new tserver with Master: bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487)
I20240503 14:12:00.169317 20918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.8:48573
I20240503 14:12:00.174804 20876 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:12:00.196640 20876 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:12:00.196815 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21500
I20240503 14:12:00.207123 21622 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.220113 21757 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
I20240503 14:12:00.290890 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21500
I20240503 14:12:00.307493 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21630
I20240503 14:12:00.317375 21752 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.399894 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21630
I20240503 14:12:00.423067 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21760
I20240503 14:12:00.431411 21882 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.512368 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21760
I20240503 14:12:00.528298 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21890
I20240503 14:12:00.536597 22012 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.618961 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21890
I20240503 14:12:04.389400 20918 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:04.436751 21301 tablet_service.cc:1467] Processing CreateTablet for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.436751 21172 tablet_service.cc:1467] Processing CreateTablet for tablet de7258a94fee4aa49a19fe1436526abd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.438330 21301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.438330 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7258a94fee4aa49a19fe1436526abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.439976 21041 tablet_service.cc:1467] Processing CreateTablet for tablet de7258a94fee4aa49a19fe1436526abd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.440721 21171 tablet_service.cc:1467] Processing CreateTablet for tablet 8675b2b41d0d4058b286e73a43751a48 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.441212 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8675b2b41d0d4058b286e73a43751a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.441394 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7258a94fee4aa49a19fe1436526abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.442835 21039 tablet_service.cc:1467] Processing CreateTablet for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.443324 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.444222 21302 tablet_service.cc:1467] Processing CreateTablet for tablet de7258a94fee4aa49a19fe1436526abd (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.444485 21040 tablet_service.cc:1467] Processing CreateTablet for tablet 8675b2b41d0d4058b286e73a43751a48 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.445039 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7258a94fee4aa49a19fe1436526abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.445420 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8675b2b41d0d4058b286e73a43751a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.446324 21432 tablet_service.cc:1467] Processing CreateTablet for tablet 8675b2b41d0d4058b286e73a43751a48 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.447742 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8675b2b41d0d4058b286e73a43751a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.459318 21431 tablet_service.cc:1467] Processing CreateTablet for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:04.459867 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:04.469357 22047 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:04.474720 22049 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:04.475984 22046 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:04.478849 22049 tablet_bootstrap.cc:654] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.480185 22049 log.cc:824] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:04.480460 22046 tablet_bootstrap.cc:654] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.480713 22047 tablet_bootstrap.cc:654] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.482129 22046 log.cc:824] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:04.482393 22047 log.cc:824] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:04.484267 22048 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:04.488704 22048 tablet_bootstrap.cc:654] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.489897 22049 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:04.490057 22048 log.cc:824] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:04.490255 22049 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20240503 14:12:04.491513 22047 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:04.491866 22047 ts_tablet_manager.cc:1367] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.000s
I20240503 14:12:04.499749 22048 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:04.500125 22048 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240503 14:12:04.501030 22049 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.502326 22049 raft_consensus.cc:374] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.502498 22049 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.503348 22049 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.504242 22046 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:04.504638 22046 ts_tablet_manager.cc:1367] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.020s
I20240503 14:12:04.504593 22047 raft_consensus.cc:348] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.505108 22047 raft_consensus.cc:374] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.505249 22047 raft_consensus.cc:729] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.505978 22047 consensus_queue.cc:260] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.511546 22048 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.511889 22049 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.021s user 0.017s sys 0.002s
I20240503 14:12:04.512024 22048 raft_consensus.cc:374] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.512137 22048 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.512915 22048 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.515288 22047 ts_tablet_manager.cc:1398] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.023s user 0.021s sys 0.000s
I20240503 14:12:04.515800 22047 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:04.517004 22046 raft_consensus.cc:348] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.517498 22046 raft_consensus.cc:374] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.517645 22047 tablet_bootstrap.cc:654] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.517617 22046 raft_consensus.cc:729] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.518445 22046 consensus_queue.cc:260] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.520856 22049 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:04.522306 22048 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.022s user 0.011s sys 0.004s
I20240503 14:12:04.522532 22049 tablet_bootstrap.cc:654] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.522883 22048 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:04.526559 22048 tablet_bootstrap.cc:654] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.531601 22046 ts_tablet_manager.cc:1398] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.027s user 0.010s sys 0.014s
I20240503 14:12:04.532066 22046 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:04.532199 22047 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:04.532203 22048 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:04.532346 22047 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20240503 14:12:04.532346 22048 ts_tablet_manager.cc:1367] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:12:04.533353 22047 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.533586 22047 raft_consensus.cc:374] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.533676 22047 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.533784 22046 tablet_bootstrap.cc:654] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.533902 22047 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.534466 22047 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:04.534909 22049 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:04.535115 22049 ts_tablet_manager.cc:1367] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20240503 14:12:04.537006 22049 raft_consensus.cc:348] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.537288 22049 raft_consensus.cc:374] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.537391 22049 raft_consensus.cc:729] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.537639 22048 raft_consensus.cc:348] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.537859 22048 raft_consensus.cc:374] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.537957 22048 raft_consensus.cc:729] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.537964 22049 consensus_queue.cc:260] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.538259 22048 consensus_queue.cc:260] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.538878 22048 ts_tablet_manager.cc:1398] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:12:04.539474 22049 ts_tablet_manager.cc:1398] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:04.539811 22049 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:04.540650 22046 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:04.540951 22046 ts_tablet_manager.cc:1367] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240503 14:12:04.541584 22049 tablet_bootstrap.cc:654] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:04.541987 22046 raft_consensus.cc:348] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.542199 22046 raft_consensus.cc:374] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.542291 22046 raft_consensus.cc:729] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.542593 22046 consensus_queue.cc:260] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.543268 22046 ts_tablet_manager.cc:1398] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:04.544449 22049 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:04.544593 22049 ts_tablet_manager.cc:1367] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:12:04.545284 22049 raft_consensus.cc:348] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.545504 22049 raft_consensus.cc:374] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:04.545590 22049 raft_consensus.cc:729] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:04.545830 22049 consensus_queue.cc:260] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.546293 22049 ts_tablet_manager.cc:1398] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240503 14:12:04.548775 21368 tablet.cc:2369] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:04.569363 21238 tablet.cc:2369] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:04.601933 22054 raft_consensus.cc:482] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.602262 22054 raft_consensus.cc:504] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.604190 22054 leader_election.cc:288] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
W20240503 14:12:04.608723 21107 tablet.cc:2369] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:04.613042 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:04.613507 21322 raft_consensus.cc:2454] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 0.
I20240503 14:12:04.614104 20996 leader_election.cc:302] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:04.614212 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:04.614557 22054 raft_consensus.cc:2781] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:04.614668 22054 raft_consensus.cc:482] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:04.614683 21192 raft_consensus.cc:2454] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 0.
I20240503 14:12:04.614729 22054 raft_consensus.cc:3037] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.616849 22054 raft_consensus.cc:504] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.617389 22054 leader_election.cc:288] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:04.617794 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:04.617838 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:04.617942 21192 raft_consensus.cc:3037] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.618007 21322 raft_consensus.cc:3037] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.620038 21192 raft_consensus.cc:2454] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 1.
I20240503 14:12:04.620071 21322 raft_consensus.cc:2454] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 1.
I20240503 14:12:04.620465 20996 leader_election.cc:302] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:04.620762 22054 raft_consensus.cc:2781] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:04.621218 22054 raft_consensus.cc:686] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 1 LEADER]: Becoming Leader. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Running, Role: LEADER
I20240503 14:12:04.621721 22054 consensus_queue.cc:237] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:04.626303 20919 catalog_manager.cc:5484] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1114934d9a3544a9935d75504bc27ceb (127.20.99.1). New cstate: current_term: 1 leader_uuid: "1114934d9a3544a9935d75504bc27ceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:04.641388 22055 raft_consensus.cc:482] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.641644 22055 raft_consensus.cc:504] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.643390 22055 leader_election.cc:288] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:04.651683 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:04.651973 21322 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:04.652241 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:04.652487 21387 leader_election.cc:302] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:04.652725 21061 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:04.652863 22055 raft_consensus.cc:2781] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:04.653002 22055 raft_consensus.cc:482] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:04.653131 22055 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.655090 22055 raft_consensus.cc:504] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.655740 22055 leader_election.cc:288] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Requested vote from peers bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:04.656067 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:04.656219 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:04.656306 21322 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.656405 21061 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.658358 21322 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:04.658551 21061 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:04.658705 21387 leader_election.cc:302] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:04.658957 22055 raft_consensus.cc:2781] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:04.659497 22055 raft_consensus.cc:686] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:04.659986 22055 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.664484 20919 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:04.763689 21498 tablet.cc:2369] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:04.781533 22054 raft_consensus.cc:482] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:04.781776 22054 raft_consensus.cc:504] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.782541 22054 leader_election.cc:288] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:04.783161 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:04.783453 21192 raft_consensus.cc:2454] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 0.
I20240503 14:12:04.783869 20995 leader_election.cc:302] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:04.784128 22054 raft_consensus.cc:2781] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:04.784204 22054 raft_consensus.cc:482] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:04.784313 22054 raft_consensus.cc:3037] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.786265 22054 raft_consensus.cc:504] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.786906 22054 leader_election.cc:288] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:04.787242 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:04.787456 21192 raft_consensus.cc:3037] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.788637 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:04.788662 21452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
I20240503 14:12:04.789170 21451 raft_consensus.cc:3037] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:04.790160 21192 raft_consensus.cc:2454] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 1.
I20240503 14:12:04.790539 20995 leader_election.cc:302] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:04.790848 22054 raft_consensus.cc:2781] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:04.791126 22054 raft_consensus.cc:686] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 1 LEADER]: Becoming Leader. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Running, Role: LEADER
I20240503 14:12:04.791435 22054 consensus_queue.cc:237] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:04.792173 21451 raft_consensus.cc:2454] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 1.
I20240503 14:12:04.794278 20919 catalog_manager.cc:5484] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1114934d9a3544a9935d75504bc27ceb (127.20.99.1). New cstate: current_term: 1 leader_uuid: "1114934d9a3544a9935d75504bc27ceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:05.074326 22055 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.081985 22075 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.127919 22054 consensus_queue.cc:1035] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.134110 22054 consensus_queue.cc:1035] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:05.144145 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:05.144493 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:05.152294 22072 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:12:05.182674 22054 consensus_queue.cc:1035] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:05.188402 22054 consensus_queue.cc:1035] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:05.439311 20996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:43159, user_credentials={real_user=slave}} blocked reactor thread for 50119.6us
W20240503 14:12:05.443802 20995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:33731, user_credentials={real_user=slave}} blocked reactor thread for 59329.7us
INSERT report
rows total: 2000
time total: 280.91 ms
time per row: 0.140455 ms
I20240503 14:12:05.575858 20910 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:05.584493 21171 tablet_service.cc:1467] Processing CreateTablet for tablet a955820c3a79415c9d118901eb40f172 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.584911 21431 tablet_service.cc:1467] Processing CreateTablet for tablet a955820c3a79415c9d118901eb40f172 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.585065 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a955820c3a79415c9d118901eb40f172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.585388 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a955820c3a79415c9d118901eb40f172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.586597 21039 tablet_service.cc:1467] Processing CreateTablet for tablet 98f8323a5a1c45ec97a4ccbc937fb135 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.587015 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f8323a5a1c45ec97a4ccbc937fb135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.587023 21432 tablet_service.cc:1467] Processing CreateTablet for tablet 2f2c7555d83d4300b8b99e186e92a794 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.587421 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c7555d83d4300b8b99e186e92a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.590075 21301 tablet_service.cc:1467] Processing CreateTablet for tablet a955820c3a79415c9d118901eb40f172 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.590163 21300 tablet_service.cc:1467] Processing CreateTablet for tablet 98f8323a5a1c45ec97a4ccbc937fb135 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.590528 21301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a955820c3a79415c9d118901eb40f172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.591320 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f8323a5a1c45ec97a4ccbc937fb135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.591161 21172 tablet_service.cc:1467] Processing CreateTablet for tablet 2f2c7555d83d4300b8b99e186e92a794 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.591512 21170 tablet_service.cc:1467] Processing CreateTablet for tablet 98f8323a5a1c45ec97a4ccbc937fb135 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.591706 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c7555d83d4300b8b99e186e92a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.591962 21170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f8323a5a1c45ec97a4ccbc937fb135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.596240 21302 tablet_service.cc:1467] Processing CreateTablet for tablet 2f2c7555d83d4300b8b99e186e92a794 (DEFAULT_TABLE table=rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:05.596644 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c7555d83d4300b8b99e186e92a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:05.600487 22139 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:05.602660 22139 tablet_bootstrap.cc:654] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.605337 22140 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:05.609417 22140 tablet_bootstrap.cc:654] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.613725 22138 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:05.615123 22141 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:05.615393 22139 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:05.615566 22139 ts_tablet_manager.cc:1367] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20240503 14:12:05.616211 22139 raft_consensus.cc:348] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.616376 22139 raft_consensus.cc:374] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.616474 22139 raft_consensus.cc:729] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.616722 22139 consensus_queue.cc:260] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.617146 22141 tablet_bootstrap.cc:654] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.617270 22138 tablet_bootstrap.cc:654] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.617338 22139 ts_tablet_manager.cc:1398] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:05.617599 22139 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:05.619153 22140 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:05.619207 22139 tablet_bootstrap.cc:654] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.619351 22140 ts_tablet_manager.cc:1367] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 14:12:05.620203 22140 raft_consensus.cc:348] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.620419 22140 raft_consensus.cc:374] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.620554 22140 raft_consensus.cc:729] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.620807 22140 consensus_queue.cc:260] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.621397 22140 ts_tablet_manager.cc:1398] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:05.622184 22141 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:05.622370 22141 ts_tablet_manager.cc:1367] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240503 14:12:05.623411 22141 raft_consensus.cc:348] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.623693 22141 raft_consensus.cc:374] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.623801 22141 raft_consensus.cc:729] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.624089 22141 consensus_queue.cc:260] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.625607 22141 ts_tablet_manager.cc:1398] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:05.625875 22141 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:05.626125 22138 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:05.626273 22138 ts_tablet_manager.cc:1367] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240503 14:12:05.627099 22138 raft_consensus.cc:348] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.627349 22138 raft_consensus.cc:374] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.627468 22141 tablet_bootstrap.cc:654] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.627449 22138 raft_consensus.cc:729] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.627718 22138 consensus_queue.cc:260] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.629106 22138 ts_tablet_manager.cc:1398] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:05.629474 22138 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:05.632146 22138 tablet_bootstrap.cc:654] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.634434 22139 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:05.634608 22139 ts_tablet_manager.cc:1367] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20240503 14:12:05.635371 22139 raft_consensus.cc:348] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.635541 22138 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:05.635628 22139 raft_consensus.cc:374] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.635690 22138 ts_tablet_manager.cc:1367] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:05.635715 22139 raft_consensus.cc:729] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.635998 22141 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:05.635975 22139 consensus_queue.cc:260] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.636160 22141 ts_tablet_manager.cc:1367] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.009s
I20240503 14:12:05.636507 22138 raft_consensus.cc:348] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.636646 22139 ts_tablet_manager.cc:1398] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:05.636739 22138 raft_consensus.cc:374] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.636808 22138 raft_consensus.cc:729] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.636905 22141 raft_consensus.cc:348] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.637113 22141 raft_consensus.cc:374] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.637064 22138 consensus_queue.cc:260] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.637198 22141 raft_consensus.cc:729] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.637480 22141 consensus_queue.cc:260] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.637698 22138 ts_tablet_manager.cc:1398] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:05.637975 22138 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:05.639214 22141 ts_tablet_manager.cc:1398] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:12:05.639458 22141 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:05.639590 22138 tablet_bootstrap.cc:654] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.640960 22141 tablet_bootstrap.cc:654] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:05.642129 22138 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:05.642259 22138 ts_tablet_manager.cc:1367] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:05.643000 22138 raft_consensus.cc:348] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.643244 22138 raft_consensus.cc:374] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.643343 22138 raft_consensus.cc:729] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.643568 22138 consensus_queue.cc:260] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.644109 22138 ts_tablet_manager.cc:1398] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:05.646183 22141 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:05.646332 22141 ts_tablet_manager.cc:1367] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240503 14:12:05.647053 22141 raft_consensus.cc:348] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.647277 22141 raft_consensus.cc:374] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:05.647368 22141 raft_consensus.cc:729] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:05.647578 22141 consensus_queue.cc:260] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.648001 22141 ts_tablet_manager.cc:1398] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:05.803942 22077 raft_consensus.cc:482] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.804203 22077 raft_consensus.cc:504] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.805917 22077 leader_election.cc:288] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:05.813745 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f8323a5a1c45ec97a4ccbc937fb135" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:05.813995 21061 raft_consensus.cc:2454] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:05.814584 21256 leader_election.cc:302] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:05.814962 22077 raft_consensus.cc:2781] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.815078 22077 raft_consensus.cc:482] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.815073 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f8323a5a1c45ec97a4ccbc937fb135" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:05.815199 22077 raft_consensus.cc:3037] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.815330 21191 raft_consensus.cc:2454] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:05.817265 22077 raft_consensus.cc:504] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.817826 22077 leader_election.cc:288] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Requested vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:05.818337 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f8323a5a1c45ec97a4ccbc937fb135" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:05.818321 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98f8323a5a1c45ec97a4ccbc937fb135" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:05.818543 21191 raft_consensus.cc:3037] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.818547 21061 raft_consensus.cc:3037] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.821413 21061 raft_consensus.cc:2454] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:05.821427 21191 raft_consensus.cc:2454] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:05.821877 21256 leader_election.cc:302] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:05.822203 22077 raft_consensus.cc:2781] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.822690 22077 raft_consensus.cc:686] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Becoming Leader. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Running, Role: LEADER
I20240503 14:12:05.823215 22077 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.826721 20919 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: term changed from 0 to 1, leader changed from <none> to bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3). New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:05.851784 22055 raft_consensus.cc:482] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.851963 22055 raft_consensus.cc:504] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.852866 22055 leader_election.cc:288] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:05.854262 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a955820c3a79415c9d118901eb40f172" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:05.854512 21321 raft_consensus.cc:2454] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:05.854987 21387 leader_election.cc:302] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:05.855332 22055 raft_consensus.cc:2781] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.855458 22055 raft_consensus.cc:482] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.855540 22055 raft_consensus.cc:3037] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.857149 22075 raft_consensus.cc:482] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:05.857347 22075 raft_consensus.cc:504] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.858100 22075 leader_election.cc:288] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:05.858949 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c7555d83d4300b8b99e186e92a794" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:05.859172 21321 raft_consensus.cc:2454] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:05.859652 21387 leader_election.cc:302] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:05.859943 22075 raft_consensus.cc:2781] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:05.860050 22075 raft_consensus.cc:482] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:05.860169 22075 raft_consensus.cc:3037] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.861258 21191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a955820c3a79415c9d118901eb40f172" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:05.861441 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c7555d83d4300b8b99e186e92a794" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:05.861543 21191 raft_consensus.cc:2454] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:05.861617 21192 raft_consensus.cc:2454] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:05.870873 22055 raft_consensus.cc:504] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.871824 22055 leader_election.cc:288] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:05.872707 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a955820c3a79415c9d118901eb40f172" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:05.872715 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a955820c3a79415c9d118901eb40f172" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:05.872921 21321 raft_consensus.cc:3037] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.872951 21192 raft_consensus.cc:3037] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.895550 21192 raft_consensus.cc:2454] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:05.895535 22075 raft_consensus.cc:504] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.895817 21321 raft_consensus.cc:2454] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:05.896157 21386 leader_election.cc:302] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:05.896487 22055 raft_consensus.cc:2781] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.896893 22075 leader_election.cc:288] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:05.897595 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c7555d83d4300b8b99e186e92a794" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:05.897593 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f2c7555d83d4300b8b99e186e92a794" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:05.897822 21321 raft_consensus.cc:3037] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.897823 21192 raft_consensus.cc:3037] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:05.898381 22055 raft_consensus.cc:686] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:05.898732 22055 consensus_queue.cc:237] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:05.902208 20919 catalog_manager.cc:5484] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:05.922787 21192 raft_consensus.cc:2454] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:05.922787 21321 raft_consensus.cc:2454] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:05.923432 21387 leader_election.cc:302] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:05.923830 22055 raft_consensus.cc:2781] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:05.924010 22055 raft_consensus.cc:686] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:05.924310 22055 consensus_queue.cc:237] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:05.927090 20919 catalog_manager.cc:5484] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:06.322681 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:06.323395 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:06.335376 22160 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:06.353106 22169 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.359131 22173 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 14:12:06.381559 22055 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.387938 22075 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.447835 22075 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:06.454588 22157 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.617 ms
time per row: 0.119808 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:07.677886 20912 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:07.687023 21039 tablet_service.cc:1467] Processing CreateTablet for tablet df4f572b770c4fdf8d91d4f449f579eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.687310 21432 tablet_service.cc:1467] Processing CreateTablet for tablet df4f572b770c4fdf8d91d4f449f579eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.687552 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4f572b770c4fdf8d91d4f449f579eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.687774 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4f572b770c4fdf8d91d4f449f579eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.687935 21170 tablet_service.cc:1467] Processing CreateTablet for tablet df4f572b770c4fdf8d91d4f449f579eb (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.688333 21170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4f572b770c4fdf8d91d4f449f579eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.688638 21430 tablet_service.cc:1467] Processing CreateTablet for tablet b50c6f09335d4cb4a2c2288303fc1a7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.689002 21430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50c6f09335d4cb4a2c2288303fc1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.691552 21431 tablet_service.cc:1467] Processing CreateTablet for tablet 1dca8761a296481cbdf63afd8a4d4fa1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.691905 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dca8761a296481cbdf63afd8a4d4fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.691926 21041 tablet_service.cc:1467] Processing CreateTablet for tablet 1dca8761a296481cbdf63afd8a4d4fa1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.692276 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dca8761a296481cbdf63afd8a4d4fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.694423 21300 tablet_service.cc:1467] Processing CreateTablet for tablet 1dca8761a296481cbdf63afd8a4d4fa1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.694806 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dca8761a296481cbdf63afd8a4d4fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.695729 21171 tablet_service.cc:1467] Processing CreateTablet for tablet b50c6f09335d4cb4a2c2288303fc1a7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.696113 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50c6f09335d4cb4a2c2288303fc1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.700069 21040 tablet_service.cc:1467] Processing CreateTablet for tablet b50c6f09335d4cb4a2c2288303fc1a7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:07.700685 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50c6f09335d4cb4a2c2288303fc1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.703815 22234 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:07.706175 22234 tablet_bootstrap.cc:654] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.709430 22233 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:07.711853 22233 tablet_bootstrap.cc:654] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.712718 22231 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:07.714985 22231 tablet_bootstrap.cc:654] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.716750 22232 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:07.717985 22234 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:07.718192 22234 ts_tablet_manager.cc:1367] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20240503 14:12:07.719015 22231 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:07.719329 22231 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:12:07.719434 22234 raft_consensus.cc:348] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.719763 22234 raft_consensus.cc:374] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.719918 22234 raft_consensus.cc:729] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.719950 22232 tablet_bootstrap.cc:654] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.720276 22234 consensus_queue.cc:260] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.720481 22231 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.720746 22231 raft_consensus.cc:374] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.720849 22231 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.721101 22231 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.722294 22231 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:07.722712 22231 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:07.723297 22233 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:07.723493 22233 ts_tablet_manager.cc:1367] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240503 14:12:07.724191 22234 ts_tablet_manager.cc:1398] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:07.724365 22231 tablet_bootstrap.cc:654] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.724278 22233 raft_consensus.cc:348] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.724539 22233 raft_consensus.cc:374] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.724676 22233 raft_consensus.cc:729] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.725934 22233 consensus_queue.cc:260] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.726629 22234 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:07.727140 22233 ts_tablet_manager.cc:1398] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:07.728408 22234 tablet_bootstrap.cc:654] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.728878 22231 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:07.729028 22231 ts_tablet_manager.cc:1367] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 14:12:07.729696 22231 raft_consensus.cc:348] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.729966 22231 raft_consensus.cc:374] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.730067 22231 raft_consensus.cc:729] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.730300 22231 consensus_queue.cc:260] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.730679 22234 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:07.730888 22234 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:07.731436 22231 ts_tablet_manager.cc:1398] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:07.731825 22232 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:07.731714 22234 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.731916 22234 raft_consensus.cc:374] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.731976 22232 ts_tablet_manager.cc:1367] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240503 14:12:07.731985 22234 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.732229 22234 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.732743 22234 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:07.732723 22232 raft_consensus.cc:348] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.732916 22232 raft_consensus.cc:374] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.733003 22232 raft_consensus.cc:729] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.733075 22234 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:07.733227 22232 consensus_queue.cc:260] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.733742 22232 ts_tablet_manager.cc:1398] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:07.734050 22232 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:07.735569 22232 tablet_bootstrap.cc:654] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.737187 22234 tablet_bootstrap.cc:654] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.737500 22232 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:07.737893 22232 ts_tablet_manager.cc:1367] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:07.739737 22232 raft_consensus.cc:348] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.740041 22232 raft_consensus.cc:374] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.740208 22232 raft_consensus.cc:729] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.740753 22232 consensus_queue.cc:260] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.741205 22234 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:07.741737 22234 ts_tablet_manager.cc:1367] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 14:12:07.741858 22232 ts_tablet_manager.cc:1398] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:07.742152 22232 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:07.742498 22234 raft_consensus.cc:348] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.742697 22234 raft_consensus.cc:374] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.742820 22234 raft_consensus.cc:729] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.743058 22234 consensus_queue.cc:260] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.743697 22234 ts_tablet_manager.cc:1398] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:07.743841 22232 tablet_bootstrap.cc:654] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.745640 22232 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:07.745767 22232 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:07.746485 22232 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.746682 22232 raft_consensus.cc:374] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.746799 22232 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.747014 22232 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.747540 22232 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:07.762744 22239 raft_consensus.cc:482] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.762926 22239 raft_consensus.cc:504] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.764398 22239 leader_election.cc:288] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:07.787225 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50c6f09335d4cb4a2c2288303fc1a7f" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
I20240503 14:12:07.787520 21451 raft_consensus.cc:2454] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 0.
I20240503 14:12:07.787629 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50c6f09335d4cb4a2c2288303fc1a7f" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:07.787859 21061 raft_consensus.cc:2454] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 0.
I20240503 14:12:07.788051 21128 leader_election.cc:302] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:07.788491 22239 raft_consensus.cc:2781] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.788594 22239 raft_consensus.cc:482] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.788667 22239 raft_consensus.cc:3037] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.790571 22239 raft_consensus.cc:504] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.791158 22239 leader_election.cc:288] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 election: Requested vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:07.791657 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50c6f09335d4cb4a2c2288303fc1a7f" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:07.791719 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50c6f09335d4cb4a2c2288303fc1a7f" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:07.791849 21451 raft_consensus.cc:3037] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.791920 21061 raft_consensus.cc:3037] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.793696 21451 raft_consensus.cc:2454] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 1.
I20240503 14:12:07.794126 21128 leader_election.cc:302] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:07.794404 22239 raft_consensus.cc:2781] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.794639 21061 raft_consensus.cc:2454] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 1.
I20240503 14:12:07.794900 22239 raft_consensus.cc:686] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Running, Role: LEADER
I20240503 14:12:07.795574 22239 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.799556 20910 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:07.855700 22205 raft_consensus.cc:482] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.855893 22205 raft_consensus.cc:504] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.856647 22205 leader_election.cc:288] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:07.857235 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dca8761a296481cbdf63afd8a4d4fa1" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:07.857553 21061 raft_consensus.cc:2454] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:07.857975 21256 leader_election.cc:302] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:07.858314 22205 raft_consensus.cc:2781] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.858448 22205 raft_consensus.cc:482] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.858541 22205 raft_consensus.cc:3037] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.861421 22205 raft_consensus.cc:504] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.862531 22205 leader_election.cc:288] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Requested vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:07.862847 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dca8761a296481cbdf63afd8a4d4fa1" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:07.863090 21061 raft_consensus.cc:3037] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.863569 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dca8761a296481cbdf63afd8a4d4fa1" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
I20240503 14:12:07.863646 21452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dca8761a296481cbdf63afd8a4d4fa1" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:07.863776 21451 raft_consensus.cc:2454] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:07.865432 21061 raft_consensus.cc:2454] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:07.865783 21256 leader_election.cc:302] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters: 20560106dcaf405bbb288e4e3e8affa6
I20240503 14:12:07.866070 22205 raft_consensus.cc:2781] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.869392 22205 raft_consensus.cc:686] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Becoming Leader. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Running, Role: LEADER
I20240503 14:12:07.869689 22205 consensus_queue.cc:237] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:07.872799 20910 catalog_manager.cc:5484] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: term changed from 0 to 1, leader changed from <none> to bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3). New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:07.973187 22239 raft_consensus.cc:482] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:07.973448 22239 raft_consensus.cc:504] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.974056 22239 leader_election.cc:288] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:07.974579 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:07.974579 21452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
I20240503 14:12:07.974805 21452 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 0.
I20240503 14:12:07.974805 21061 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 0.
I20240503 14:12:07.975229 21126 leader_election.cc:302] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:07.975507 22239 raft_consensus.cc:2781] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:07.975600 22239 raft_consensus.cc:482] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:07.975708 22239 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.977770 22239 raft_consensus.cc:504] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.978308 22239 leader_election.cc:288] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 election: Requested vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:07.978722 21452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:07.978737 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:07.978883 21452 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.978893 21061 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.980914 21452 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 1.
I20240503 14:12:07.980914 21061 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 1.
I20240503 14:12:07.981304 21126 leader_election.cc:302] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:07.981587 22239 raft_consensus.cc:2781] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.981734 22239 raft_consensus.cc:686] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Running, Role: LEADER
I20240503 14:12:07.981994 22239 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:07.984614 20910 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:08.195783 22239 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.202016 22239 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:08.340876 22255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.341243 22255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.348791 22255 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:12:08.383234 21452 raft_consensus.cc:3037] T 1dca8761a296481cbdf63afd8a4d4fa1 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:08.386559 22205 consensus_queue.cc:1035] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.398744 22277 consensus_queue.cc:1035] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.437381 22239 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:08.442947 22249 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.072 ms
time per row: 0.117536 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:09.671386 20915 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:09.680662 21041 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff7de43a9a048f7941ad0143d0b09f2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.680830 21170 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff7de43a9a048f7941ad0143d0b09f2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.681185 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff7de43a9a048f7941ad0143d0b09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.681327 21170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff7de43a9a048f7941ad0143d0b09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.681604 21300 tablet_service.cc:1467] Processing CreateTablet for tablet 8ff7de43a9a048f7941ad0143d0b09f2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.681775 21301 tablet_service.cc:1467] Processing CreateTablet for tablet b8c3db38486c4d41a85551928c0f2202 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.682029 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ff7de43a9a048f7941ad0143d0b09f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.682744 21302 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbc2976e019427c915aced7b0d4a50c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.683076 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbc2976e019427c915aced7b0d4a50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.683652 21301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c3db38486c4d41a85551928c0f2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.688148 21431 tablet_service.cc:1467] Processing CreateTablet for tablet b8c3db38486c4d41a85551928c0f2202 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.688479 21432 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbc2976e019427c915aced7b0d4a50c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.688601 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c3db38486c4d41a85551928c0f2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.688817 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbc2976e019427c915aced7b0d4a50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.689044 21040 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbc2976e019427c915aced7b0d4a50c (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.689447 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbc2976e019427c915aced7b0d4a50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.694989 22324 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:09.695647 21171 tablet_service.cc:1467] Processing CreateTablet for tablet b8c3db38486c4d41a85551928c0f2202 (DEFAULT_TABLE table=rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.696095 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c3db38486c4d41a85551928c0f2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.696642 22324 tablet_bootstrap.cc:654] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.701844 22324 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:09.702558 22324 ts_tablet_manager.cc:1367] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:12:09.703701 22324 raft_consensus.cc:348] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.705050 22324 raft_consensus.cc:374] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.705340 22324 raft_consensus.cc:729] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.705785 22324 consensus_queue.cc:260] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.706836 22324 ts_tablet_manager.cc:1398] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:09.707446 22327 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:09.707648 22324 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:09.709196 22327 tablet_bootstrap.cc:654] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.709216 22324 tablet_bootstrap.cc:654] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.712783 22325 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:09.714943 22325 tablet_bootstrap.cc:654] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.716976 22326 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:09.718070 22324 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:09.718089 22325 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:09.718256 22324 ts_tablet_manager.cc:1367] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:12:09.718310 22325 ts_tablet_manager.cc:1367] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:12:09.719209 22326 tablet_bootstrap.cc:654] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.719116 22324 raft_consensus.cc:348] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.719393 22324 raft_consensus.cc:374] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.719125 22325 raft_consensus.cc:348] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.719482 22324 raft_consensus.cc:729] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.719610 22325 raft_consensus.cc:374] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.719745 22325 raft_consensus.cc:729] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.719786 22324 consensus_queue.cc:260] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.720077 22325 consensus_queue.cc:260] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.720338 22324 ts_tablet_manager.cc:1398] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:09.721784 22327 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:09.721951 22327 ts_tablet_manager.cc:1367] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20240503 14:12:09.722681 22326 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:09.722983 22326 ts_tablet_manager.cc:1367] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:12:09.723111 22325 ts_tablet_manager.cc:1398] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:09.723449 22325 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:09.724550 22326 raft_consensus.cc:348] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.724769 22326 raft_consensus.cc:374] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.724874 22326 raft_consensus.cc:729] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.725164 22326 consensus_queue.cc:260] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.722805 22327 raft_consensus.cc:348] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.725414 22327 raft_consensus.cc:374] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.725528 22325 tablet_bootstrap.cc:654] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.725545 22327 raft_consensus.cc:729] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.725847 22327 consensus_queue.cc:260] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.726702 22327 ts_tablet_manager.cc:1398] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:09.727097 22327 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:09.727241 22326 ts_tablet_manager.cc:1398] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:09.728293 22326 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:09.730765 22327 tablet_bootstrap.cc:654] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.730898 22326 tablet_bootstrap.cc:654] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.733731 22325 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:09.733960 22325 ts_tablet_manager.cc:1367] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240503 14:12:09.735004 22325 raft_consensus.cc:348] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.735353 22325 raft_consensus.cc:374] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.735453 22325 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.735785 22325 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.736543 22325 ts_tablet_manager.cc:1398] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:09.738458 22327 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:09.738622 22327 ts_tablet_manager.cc:1367] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240503 14:12:09.738399 22326 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:09.739125 22326 ts_tablet_manager.cc:1367] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20240503 14:12:09.739300 22327 raft_consensus.cc:348] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.739513 22327 raft_consensus.cc:374] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.739590 22327 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.739825 22327 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.739867 22326 raft_consensus.cc:348] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.740336 22327 ts_tablet_manager.cc:1398] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:09.740693 22326 raft_consensus.cc:374] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.740779 22326 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.740985 22326 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.742094 22326 ts_tablet_manager.cc:1398] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:09.742388 22326 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:09.746665 22326 tablet_bootstrap.cc:654] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.750207 22326 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:09.750324 22326 ts_tablet_manager.cc:1367] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:12:09.750857 22326 raft_consensus.cc:348] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.751008 22326 raft_consensus.cc:374] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.751070 22326 raft_consensus.cc:729] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.751274 22326 consensus_queue.cc:260] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.751703 22326 ts_tablet_manager.cc:1398] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:12:09.782603 22249 raft_consensus.cc:482] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.782826 22249 raft_consensus.cc:504] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.783583 22249 leader_election.cc:288] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:09.784236 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff7de43a9a048f7941ad0143d0b09f2" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:09.784562 21061 raft_consensus.cc:2454] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 0.
I20240503 14:12:09.785039 21126 leader_election.cc:302] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:09.785391 22249 raft_consensus.cc:2781] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.785523 22249 raft_consensus.cc:482] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.785668 22249 raft_consensus.cc:3037] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.788061 22249 raft_consensus.cc:504] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.788816 22249 leader_election.cc:288] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 election: Requested vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:09.789426 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff7de43a9a048f7941ad0143d0b09f2" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:09.789898 21061 raft_consensus.cc:3037] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.791209 21320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff7de43a9a048f7941ad0143d0b09f2" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:09.791493 21320 raft_consensus.cc:2454] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 0.
I20240503 14:12:09.791500 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ff7de43a9a048f7941ad0143d0b09f2" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:09.792379 21321 raft_consensus.cc:3037] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.792500 21061 raft_consensus.cc:2454] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 1.
I20240503 14:12:09.792933 21126 leader_election.cc:302] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:09.793238 22249 raft_consensus.cc:2781] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.795243 21321 raft_consensus.cc:2454] T 8ff7de43a9a048f7941ad0143d0b09f2 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 1.
I20240503 14:12:09.799836 22249 raft_consensus.cc:686] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Running, Role: LEADER
I20240503 14:12:09.800148 22249 consensus_queue.cc:237] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:09.802763 20915 catalog_manager.cc:5484] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:09.914897 22157 raft_consensus.cc:482] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.915117 22157 raft_consensus.cc:504] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.915832 22157 leader_election.cc:288] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:09.916321 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:09.916445 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:09.916555 21321 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:09.916682 21192 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:09.916980 21387 leader_election.cc:302] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:09.917275 22157 raft_consensus.cc:2781] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.917357 22157 raft_consensus.cc:482] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.917474 22157 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.919349 22157 raft_consensus.cc:504] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.920095 22157 leader_election.cc:288] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Requested vote from peers bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:09.920500 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:09.920578 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:09.920729 21321 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.920761 21192 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.923600 21321 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:09.923614 21192 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:09.924000 21387 leader_election.cc:302] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:09.924252 22157 raft_consensus.cc:2781] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.924625 22157 raft_consensus.cc:686] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:09.924854 22157 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:09.927552 20915 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:09.950244 22157 raft_consensus.cc:482] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:09.950479 22157 raft_consensus.cc:504] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.951146 22157 leader_election.cc:288] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:09.951787 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbc2976e019427c915aced7b0d4a50c" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:09.951787 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbc2976e019427c915aced7b0d4a50c" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
I20240503 14:12:09.952025 21321 raft_consensus.cc:2454] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:09.952025 21061 raft_consensus.cc:2454] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 0.
I20240503 14:12:09.952419 21386 leader_election.cc:302] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 20560106dcaf405bbb288e4e3e8affa6; no voters:
I20240503 14:12:09.952816 22157 raft_consensus.cc:2781] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:09.952968 22157 raft_consensus.cc:482] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:09.953068 22157 raft_consensus.cc:3037] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.955461 22157 raft_consensus.cc:504] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.956096 22157 leader_election.cc:288] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Requested vote from peers bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:09.956517 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbc2976e019427c915aced7b0d4a50c" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:09.956507 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbc2976e019427c915aced7b0d4a50c" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:09.956712 21061 raft_consensus.cc:3037] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.956712 21321 raft_consensus.cc:3037] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.959551 21321 raft_consensus.cc:2454] T 4cbc2976e019427c915aced7b0d4a50c P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:09.959551 21061 raft_consensus.cc:2454] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:09.960011 21386 leader_election.cc:302] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 20560106dcaf405bbb288e4e3e8affa6; no voters:
I20240503 14:12:09.960320 22157 raft_consensus.cc:2781] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.960461 22157 raft_consensus.cc:686] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:09.960745 22157 consensus_queue.cc:237] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:09.963676 20915 catalog_manager.cc:5484] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:10.193565 22249 consensus_queue.cc:1035] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.199435 22249 consensus_queue.cc:1035] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:10.327085 22344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:10.327487 22344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:10.335130 22344 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:12:10.451483 22157 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.460541 22341 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.482895 22341 consensus_queue.cc:1035] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.529714 22341 consensus_queue.cc:1035] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:10.586323 22380 mvcc.cc:205] Tried to move back new op lower bound from 7023597693168885760 to 7023597690574307328. Current Snapshot: MvccSnapshot[applied={T|T < 7023597693168885760}]
INSERT report
rows total: 2000
time total: 250.187 ms
time per row: 0.125094 ms
I20240503 14:12:10.727943 20917 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:10.736881 21300 tablet_service.cc:1467] Processing CreateTablet for tablet 943f587dd47e486f9be763a1c52bf43f (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.736964 21170 tablet_service.cc:1467] Processing CreateTablet for tablet 943f587dd47e486f9be763a1c52bf43f (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.737401 21170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943f587dd47e486f9be763a1c52bf43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.737394 21300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943f587dd47e486f9be763a1c52bf43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.738270 21041 tablet_service.cc:1467] Processing CreateTablet for tablet 943f587dd47e486f9be763a1c52bf43f (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.738596 21431 tablet_service.cc:1467] Processing CreateTablet for tablet bd6b3d66deae41439fcbf13cb5400045 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.738695 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943f587dd47e486f9be763a1c52bf43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.738983 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6b3d66deae41439fcbf13cb5400045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.743647 21432 tablet_service.cc:1467] Processing CreateTablet for tablet 39aaa8ba3e0c43078f94e265a961fe97 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.744099 21432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39aaa8ba3e0c43078f94e265a961fe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.744501 21040 tablet_service.cc:1467] Processing CreateTablet for tablet bd6b3d66deae41439fcbf13cb5400045 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.744869 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6b3d66deae41439fcbf13cb5400045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.744941 21171 tablet_service.cc:1467] Processing CreateTablet for tablet bd6b3d66deae41439fcbf13cb5400045 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.745332 21171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6b3d66deae41439fcbf13cb5400045. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.748090 21302 tablet_service.cc:1467] Processing CreateTablet for tablet 39aaa8ba3e0c43078f94e265a961fe97 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.748634 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39aaa8ba3e0c43078f94e265a961fe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.749776 21172 tablet_service.cc:1467] Processing CreateTablet for tablet 39aaa8ba3e0c43078f94e265a961fe97 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:10.750131 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39aaa8ba3e0c43078f94e265a961fe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:10.755430 22410 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:10.759099 22410 tablet_bootstrap.cc:654] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.764550 22412 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:10.765198 22411 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:10.766590 22410 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:10.767078 22412 tablet_bootstrap.cc:654] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.767194 22414 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:10.767403 22410 ts_tablet_manager.cc:1367] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240503 14:12:10.767434 22411 tablet_bootstrap.cc:654] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.768466 22410 raft_consensus.cc:348] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.768786 22410 raft_consensus.cc:374] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.768890 22410 raft_consensus.cc:729] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.769193 22410 consensus_queue.cc:260] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.771106 22410 ts_tablet_manager.cc:1398] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:10.772006 22410 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Bootstrap starting.
I20240503 14:12:10.772017 22412 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:10.772043 22411 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:10.772167 22412 ts_tablet_manager.cc:1367] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:12:10.772346 22411 ts_tablet_manager.cc:1367] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:12:10.773043 22412 raft_consensus.cc:348] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.773298 22412 raft_consensus.cc:374] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.773197 22411 raft_consensus.cc:348] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.773427 22412 raft_consensus.cc:729] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.773540 22411 raft_consensus.cc:374] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.773720 22411 raft_consensus.cc:729] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.773700 22412 consensus_queue.cc:260] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.773986 22414 tablet_bootstrap.cc:654] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.774055 22411 consensus_queue.cc:260] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.774425 22412 ts_tablet_manager.cc:1398] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.774756 22412 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:12:10.774736 22411 ts_tablet_manager.cc:1398] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.775058 22411 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: Bootstrap starting.
I20240503 14:12:10.776312 22412 tablet_bootstrap.cc:654] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.776952 22411 tablet_bootstrap.cc:654] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.778427 22410 tablet_bootstrap.cc:654] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.783525 22410 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: No bootstrap required, opened a new log
I20240503 14:12:10.783712 22410 ts_tablet_manager.cc:1367] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240503 14:12:10.785459 22410 raft_consensus.cc:348] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.785691 22410 raft_consensus.cc:374] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.785778 22410 raft_consensus.cc:729] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.786128 22410 consensus_queue.cc:260] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.788079 22410 ts_tablet_manager.cc:1398] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:10.790024 22411 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: No bootstrap required, opened a new log
I20240503 14:12:10.790067 22412 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: No bootstrap required, opened a new log
I20240503 14:12:10.790215 22411 ts_tablet_manager.cc:1367] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20240503 14:12:10.790231 22412 ts_tablet_manager.cc:1367] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20240503 14:12:10.790886 22414 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:10.791072 22414 ts_tablet_manager.cc:1367] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20240503 14:12:10.790999 22411 raft_consensus.cc:348] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.790999 22412 raft_consensus.cc:348] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.791260 22411 raft_consensus.cc:374] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.791260 22412 raft_consensus.cc:374] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.791347 22412 raft_consensus.cc:729] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.791347 22411 raft_consensus.cc:729] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.791600 22411 consensus_queue.cc:260] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.791599 22412 consensus_queue.cc:260] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.792198 22411 ts_tablet_manager.cc:1398] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:12:10.792912 22412 ts_tablet_manager.cc:1398] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:10.793285 22414 raft_consensus.cc:348] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.793525 22414 raft_consensus.cc:374] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.793632 22414 raft_consensus.cc:729] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.793849 22414 consensus_queue.cc:260] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.794416 22414 ts_tablet_manager.cc:1398] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:12:10.794734 22414 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:10.796458 22414 tablet_bootstrap.cc:654] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.798599 22414 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:10.798938 22414 ts_tablet_manager.cc:1367] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 14:12:10.799676 22414 raft_consensus.cc:348] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.799896 22414 raft_consensus.cc:374] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.799969 22414 raft_consensus.cc:729] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.800232 22414 consensus_queue.cc:260] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.800850 22414 ts_tablet_manager.cc:1398] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:10.801100 22414 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: Bootstrap starting.
I20240503 14:12:10.802726 22414 tablet_bootstrap.cc:654] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:10.803326 22303 raft_consensus.cc:482] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.803494 22303 raft_consensus.cc:504] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.804327 22303 leader_election.cc:288] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:10.805016 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943f587dd47e486f9be763a1c52bf43f" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:10.805042 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943f587dd47e486f9be763a1c52bf43f" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb" is_pre_election: true
I20240503 14:12:10.805254 21192 raft_consensus.cc:2454] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:10.805279 21061 raft_consensus.cc:2454] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:10.805634 22414 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: No bootstrap required, opened a new log
I20240503 14:12:10.805658 21256 leader_election.cc:302] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:10.805775 22414 ts_tablet_manager.cc:1367] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:10.805974 22303 raft_consensus.cc:2781] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.806085 22303 raft_consensus.cc:482] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.806174 22303 raft_consensus.cc:3037] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.806605 22414 raft_consensus.cc:348] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.806792 22414 raft_consensus.cc:374] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:10.806876 22414 raft_consensus.cc:729] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Initialized, Role: FOLLOWER
I20240503 14:12:10.808557 22414 consensus_queue.cc:260] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.808946 22303 raft_consensus.cc:504] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.809262 22414 ts_tablet_manager.cc:1398] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:10.810070 21061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943f587dd47e486f9be763a1c52bf43f" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1114934d9a3544a9935d75504bc27ceb"
I20240503 14:12:10.809604 22303 leader_election.cc:288] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:10.810298 21061 raft_consensus.cc:3037] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.811295 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943f587dd47e486f9be763a1c52bf43f" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:10.811494 21192 raft_consensus.cc:3037] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.812650 21061 raft_consensus.cc:2454] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:10.813060 21256 leader_election.cc:302] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:10.813344 22303 raft_consensus.cc:2781] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.813848 22303 raft_consensus.cc:686] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Becoming Leader. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Running, Role: LEADER
I20240503 14:12:10.814163 22303 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:12:10.814685 21192 raft_consensus.cc:2454] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:10.817683 20915 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: term changed from 0 to 1, leader changed from <none> to bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3). New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:10.849877 22303 raft_consensus.cc:482] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.850116 22303 raft_consensus.cc:504] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.850795 22303 leader_election.cc:288] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:10.851379 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
I20240503 14:12:10.851473 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:10.851604 21451 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:10.851724 21192 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 0.
I20240503 14:12:10.852123 21256 leader_election.cc:302] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:10.852438 22303 raft_consensus.cc:2781] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.852540 22303 raft_consensus.cc:482] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.852653 22303 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.855475 22303 raft_consensus.cc:504] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.856242 22303 leader_election.cc:288] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Requested vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:10.856566 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:10.856781 21451 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.856810 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:10.857009 21192 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.859218 21192 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:10.859556 21451 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:10.859673 21256 leader_election.cc:302] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
I20240503 14:12:10.859993 22303 raft_consensus.cc:2781] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.860152 22303 raft_consensus.cc:686] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Becoming Leader. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Running, Role: LEADER
I20240503 14:12:10.860435 22303 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:10.863234 20915 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: term changed from 0 to 1, leader changed from <none> to bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3). New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:10.978503 22123 raft_consensus.cc:482] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.978708 22123 raft_consensus.cc:504] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.979513 22123 leader_election.cc:288] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:10.980113 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:10.980118 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
I20240503 14:12:10.980415 21192 raft_consensus.cc:2454] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 0.
I20240503 14:12:10.980459 21451 raft_consensus.cc:2454] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 0.
I20240503 14:12:10.980978 20997 leader_election.cc:302] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 20560106dcaf405bbb288e4e3e8affa6; no voters:
I20240503 14:12:10.981282 22123 raft_consensus.cc:2781] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.981420 22123 raft_consensus.cc:482] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.981508 22123 raft_consensus.cc:3037] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.983410 22123 raft_consensus.cc:504] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.983968 22123 leader_election.cc:288] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:10.984448 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:10.984701 21192 raft_consensus.cc:3037] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.984611 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "1114934d9a3544a9935d75504bc27ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:10.984855 21451 raft_consensus.cc:3037] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.987596 21451 raft_consensus.cc:2454] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 1.
I20240503 14:12:10.987598 21192 raft_consensus.cc:2454] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1114934d9a3544a9935d75504bc27ceb in term 1.
I20240503 14:12:10.988018 20995 leader_election.cc:302] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1114934d9a3544a9935d75504bc27ceb, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:10.988417 22123 raft_consensus.cc:2781] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:10.988875 22123 raft_consensus.cc:686] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 1 LEADER]: Becoming Leader. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Running, Role: LEADER
I20240503 14:12:10.989404 22123 consensus_queue.cc:237] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:10.992223 20915 catalog_manager.cc:5484] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb reported cstate change: term changed from 0 to 1, leader changed from <none> to 1114934d9a3544a9935d75504bc27ceb (127.20.99.1). New cstate: current_term: 1 leader_uuid: "1114934d9a3544a9935d75504bc27ceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:11.356256 22303 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.364306 22303 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:11.413751 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:11.414070 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:11.421475 22428 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:11.433066 22423 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.437498 22303 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 14:12:11.491760 22123 consensus_queue.cc:1035] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:11.500649 22462 consensus_queue.cc:1035] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 209.681 ms
time per row: 0.104841 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:12.728615 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:40925
--local_ip_for_outbound_sockets=127.20.99.5
--tserver_master_addrs=127.20.99.62:39349
--webserver_port=38637
--webserver_interface=127.20.99.5
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 14:12:13.065709 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.066041 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.066210 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.074159 22489 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.074278 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.074445 22489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:12:13.083809 22489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:40925
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=38637
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:13.085227 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:13.086984 22489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:13.100597 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.101393 22495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.101972 22497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:13.103106 22489 server_base.cc:1034] running on GCE node
I20240503 14:12:13.104017 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:13.105219 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:13.106457 22489 hybrid_clock.cc:648] HybridClock initialized: now 1714745533106404 us; error 47 us; skew 500 ppm
I20240503 14:12:13.106806 22489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:13.109413 22489 webserver.cc:460] Webserver started at http://127.20.99.5:38637/ using document root <none> and password file <none>
I20240503 14:12:13.110189 22489 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:13.110319 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:13.115898 22489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:13.119263 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.120697 22489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:13.120852 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "eb49440048bd4d749779867fa4ae1489"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-p0gh"
I20240503 14:12:13.121696 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.146018 22489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.146849 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.147171 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.148154 22489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:13.149694 22489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:13.149775 22489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.149924 22489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:13.149981 22489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.180151 22489 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:40925
I20240503 14:12:13.180212 22615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:40925 every 8 connection(s)
I20240503 14:12:13.181742 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:13.186303 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 22489
I20240503 14:12:13.187086 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.6:36299
--local_ip_for_outbound_sockets=127.20.99.6
--tserver_master_addrs=127.20.99.62:39349
--webserver_port=33677
--webserver_interface=127.20.99.6
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:12:13.197500 22616 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:12:13.197810 22616 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:13.198644 22616 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:12:13.200433 20918 ts_manager.cc:194] Re-registered known tserver with Master: eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925)
I20240503 14:12:13.201725 20918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:52231
W20240503 14:12:13.525592 22620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.525882 22620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.525988 22620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.534184 22620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.534308 22620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.534438 22620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.6
I20240503 14:12:13.543928 22620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.6:36299
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.99.6
--webserver_port=33677
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.6
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:13.545272 22620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:13.546972 22620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:13.559872 22626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.560712 22627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.561465 22629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:13.562274 22620 server_base.cc:1034] running on GCE node
I20240503 14:12:13.562826 22620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:13.563930 22620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:13.565121 22620 hybrid_clock.cc:648] HybridClock initialized: now 1714745533565085 us; error 33 us; skew 500 ppm
I20240503 14:12:13.565466 22620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:13.567800 22620 webserver.cc:460] Webserver started at http://127.20.99.6:33677/ using document root <none> and password file <none>
I20240503 14:12:13.568509 22620 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:13.568637 22620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:13.574306 22620 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:12:13.577075 22635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.578461 22620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:13.578615 22620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "748b2b25b1b94a1b94788e8ece4ecb6d"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:12:13.579499 22620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.596998 22620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.597903 22620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.598212 22620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.599342 22620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:13.600821 22620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:13.600889 22620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.600979 22620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:13.601020 22620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.634737 22620 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.6:36299
I20240503 14:12:13.634774 22747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.6:36299 every 8 connection(s)
I20240503 14:12:13.636230 22620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:13.636848 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 22620
I20240503 14:12:13.637650 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.7:43693
--local_ip_for_outbound_sockets=127.20.99.7
--tserver_master_addrs=127.20.99.62:39349
--webserver_port=43911
--webserver_interface=127.20.99.7
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:12:13.649698 22748 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:12:13.650070 22748 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:13.651245 22748 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:12:13.653239 20915 ts_manager.cc:194] Re-registered known tserver with Master: 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299)
I20240503 14:12:13.654695 20915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.6:54225
W20240503 14:12:13.981797 22751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.982101 22751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.982208 22751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.990015 22751 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.990139 22751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.990260 22751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.7
I20240503 14:12:13.999302 22751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.7:43693
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.99.7
--webserver_port=43911
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.7
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:14.000609 22751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:14.002348 22751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.014742 22757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.015769 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.016395 22760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.017184 22751 server_base.cc:1034] running on GCE node
I20240503 14:12:14.017777 22751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:14.018790 22751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:14.019984 22751 hybrid_clock.cc:648] HybridClock initialized: now 1714745534019942 us; error 39 us; skew 500 ppm
I20240503 14:12:14.020350 22751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.022643 22751 webserver.cc:460] Webserver started at http://127.20.99.7:43911/ using document root <none> and password file <none>
I20240503 14:12:14.023434 22751 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.023561 22751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.028812 22751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:12:14.031597 22766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.032932 22751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:12:14.033077 22751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "ad5e6850bbff47fd9ff2e14de5b48223"
format_stamp: "Formatted at 2024-05-03 14:11:59 on dist-test-slave-p0gh"
I20240503 14:12:14.033879 22751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.074230 22751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.075138 22751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:14.075474 22751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.076519 22751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.078052 22751 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.078130 22751 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.078262 22751 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.078316 22751 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.111454 22751 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.7:43693
I20240503 14:12:14.111515 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.7:43693 every 8 connection(s)
I20240503 14:12:14.112972 22751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:14.116271 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 22751
I20240503 14:12:14.116956 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.8:40487
--local_ip_for_outbound_sockets=127.20.99.8
--tserver_master_addrs=127.20.99.62:39349
--webserver_port=41365
--webserver_interface=127.20.99.8
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 14:12:14.128885 22879 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:12:14.129235 22879 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.130179 22879 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:12:14.131785 20915 ts_manager.cc:194] Re-registered known tserver with Master: ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693)
I20240503 14:12:14.132586 20915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.7:36761
I20240503 14:12:14.204664 22616 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
W20240503 14:12:14.464155 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.464476 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:14.464622 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:14.472323 22882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:14.472481 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:14.472656 22882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.8
I20240503 14:12:14.482460 22882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.8:40487
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.99.8
--webserver_port=41365
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.8
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:14.483893 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:14.485694 22882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.498606 22889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.499462 22891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.500301 22888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.501123 22882 server_base.cc:1034] running on GCE node
I20240503 14:12:14.501744 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:14.502660 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:14.503877 22882 hybrid_clock.cc:648] HybridClock initialized: now 1714745534503826 us; error 38 us; skew 500 ppm
I20240503 14:12:14.504247 22882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.506603 22882 webserver.cc:460] Webserver started at http://127.20.99.8:41365/ using document root <none> and password file <none>
I20240503 14:12:14.507362 22882 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.507498 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.513077 22882 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:14.516252 22897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.518218 22882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:14.518406 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bb8d27ca78834bacb9ef4a67a829b494"
format_stamp: "Formatted at 2024-05-03 14:12:00 on dist-test-slave-p0gh"
I20240503 14:12:14.519356 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.551540 22882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.552423 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:14.552702 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.553659 22882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.555145 22882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.555255 22882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.555388 22882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.555445 22882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.588421 22882 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.8:40487
I20240503 14:12:14.588490 23009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.8:40487 every 8 connection(s)
I20240503 14:12:14.589886 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:12:14.598204 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 22882
I20240503 14:12:14.603065 23010 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:12:14.603432 23010 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.604547 23010 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:12:14.606076 20918 ts_manager.cc:194] Re-registered known tserver with Master: bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487)
I20240503 14:12:14.606798 20918 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.8:54891
I20240503 14:12:14.659425 22748 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
I20240503 14:12:15.095851 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:15.108920 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:15.110700 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:15.128067 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:15.132620 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:15.132611 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:15.135227 22879 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
I20240503 14:12:15.147758 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:15.150758 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:15.609242 23010 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
I20240503 14:12:15.718605 20918 ts_manager.cc:292] Set tserver state for 1114934d9a3544a9935d75504bc27ceb to MAINTENANCE_MODE
W20240503 14:12:16.150107 23077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.150425 23077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.158308 23077 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:16.396664 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:16.397536 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:16.413439 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:16.445369 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.449273 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:16.453572 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:16.470477 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.499900 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.518275 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.555248 21300 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:16.559002 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:16.596796 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.600383 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.606024 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:16.607806 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.647720 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.880211 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.880650 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:16.883420 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.899503 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.908387 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:16.908576 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:16.917383 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:16.946687 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:17.177615 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:17.185163 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:17.199612 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:17.220145 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.251237 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.276755 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.313022 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:17.342383 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.417239 21320 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } }
I20240503 14:12:17.419668 21060 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:17.420039 21191 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:17.420414 23249 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:17.420774 22752 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:17.423746 23249 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } }
I20240503 14:12:17.424332 21060 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } }
I20240503 14:12:17.424932 21191 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } }
I20240503 14:12:17.430369 20918 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 14:12:17.448361 21320 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
W20240503 14:12:17.449977 23207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:17.450522 23207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:17.451560 21191 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:17.452088 21060 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:17.453052 23249 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:17.453390 22752 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:17.456871 23249 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:17.460711 21060 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:17.461877 20918 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:17.465488 23207 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:17.465039 21191 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:17.467880 20905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 943f587dd47e486f9be763a1c52bf43f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 14:12:17.475270 21255 consensus_peers.cc:485] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b -> Peer ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693): Couldn't send request to peer ad5e6850bbff47fd9ff2e14de5b48223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 943f587dd47e486f9be763a1c52bf43f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:17.637059 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:17.639856 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.651335 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:17.653816 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:17.662050 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.663909 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:17.680595 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.707490 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:17.943163 23307 ts_tablet_manager.cc:902] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Initiating tablet copy from peer bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:17.945042 23307 tablet_copy_client.cc:308] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43159
I20240503 14:12:17.975720 21342 tablet_copy_service.cc:140] P bce0dba863d04bcc8448b0d9f3ec710b: Received BeginTabletCopySession request for tablet 943f587dd47e486f9be763a1c52bf43f from peer ad5e6850bbff47fd9ff2e14de5b48223 ({username='slave'} at 127.20.99.7:37157)
I20240503 14:12:17.976004 21342 tablet_copy_service.cc:161] P bce0dba863d04bcc8448b0d9f3ec710b: Beginning new tablet copy session on tablet 943f587dd47e486f9be763a1c52bf43f from peer ad5e6850bbff47fd9ff2e14de5b48223 at {username='slave'} at 127.20.99.7:37157: session id = ad5e6850bbff47fd9ff2e14de5b48223-943f587dd47e486f9be763a1c52bf43f
I20240503 14:12:17.980404 21342 tablet_copy_source_session.cc:205] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:17.985531 23307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943f587dd47e486f9be763a1c52bf43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.995834 23307 tablet_copy_client.cc:783] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:17.996165 23307 tablet_copy_client.cc:647] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:17.998469 23311 tablet_copy_client.cc:1002] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:18.246239 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:18.246585 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:18.254328 23306 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:18.450971 21041 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:18.461949 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:18.491452 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:18.521435 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:18.572688 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:18.572778 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:18.582314 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:18.613634 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240503 14:12:18.804822 23076 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 20979
I20240503 14:12:18.812584 21100 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:18.945677 23076 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 20979
W20240503 14:12:18.968519 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.99.1:43389 (error 108)
W20240503 14:12:18.969225 21387 connection.cc:535] server connection from 127.20.99.1:60541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:12:18.971809 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.023164 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:19.024725 21256 consensus_peers.cc:485] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.262068 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.265640 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:19.267145 21386 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.314793 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.393190 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:19.483816 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:19.502369 23307 tablet_copy_client.cc:523] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:12:19.502413 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:19.505950 23307 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap starting.
I20240503 14:12:19.515347 23307 log.cc:824] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:19.549849 23307 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:19.550366 23307 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap complete.
I20240503 14:12:19.550782 23307 ts_tablet_manager.cc:1367] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Time spent bootstrapping tablet: real 0.045s user 0.032s sys 0.012s
I20240503 14:12:19.555279 23307 raft_consensus.cc:348] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:19.555670 23307 raft_consensus.cc:729] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad5e6850bbff47fd9ff2e14de5b48223, State: Initialized, Role: LEARNER
I20240503 14:12:19.556159 23307 consensus_queue.cc:260] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:19.557544 23307 ts_tablet_manager.cc:1398] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223: Time spent starting tablet: real 0.007s user 0.009s sys 0.001s
I20240503 14:12:19.558641 21342 tablet_copy_service.cc:342] P bce0dba863d04bcc8448b0d9f3ec710b: Request end of tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-943f587dd47e486f9be763a1c52bf43f received from {username='slave'} at 127.20.99.7:37157
I20240503 14:12:19.558825 21342 tablet_copy_service.cc:433] P bce0dba863d04bcc8448b0d9f3ec710b: ending tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-943f587dd47e486f9be763a1c52bf43f on tablet 943f587dd47e486f9be763a1c52bf43f with peer ad5e6850bbff47fd9ff2e14de5b48223
I20240503 14:12:19.670526 22834 raft_consensus.cc:1203] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:12:19.944298 23326 raft_consensus.cc:482] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:19.944482 23326 raft_consensus.cc:504] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:19.945078 23326 leader_election.cc:288] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:19.945708 21452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
W20240503 14:12:19.946506 21126 leader_election.cc:334] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:19.946636 21126 leader_election.cc:302] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82; no voters: 1114934d9a3544a9935d75504bc27ceb, 20560106dcaf405bbb288e4e3e8affa6
I20240503 14:12:19.946906 23326 raft_consensus.cc:2726] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:20.021030 23326 raft_consensus.cc:482] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.021226 23326 raft_consensus.cc:504] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:20.021946 23326 leader_election.cc:288] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:20.022567 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" is_pre_election: true
W20240503 14:12:20.023326 21126 leader_election.cc:334] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.023442 21126 leader_election.cc:302] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82; no voters: 1114934d9a3544a9935d75504bc27ceb, bce0dba863d04bcc8448b0d9f3ec710b
I20240503 14:12:20.023680 23326 raft_consensus.cc:2726] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:20.223711 23326 raft_consensus.cc:482] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.223942 23326 raft_consensus.cc:504] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.224689 23326 leader_election.cc:288] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:20.225339 21452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "20560106dcaf405bbb288e4e3e8affa6" is_pre_election: true
W20240503 14:12:20.226265 21126 leader_election.cc:334] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.226403 21126 leader_election.cc:302] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82; no voters: 1114934d9a3544a9935d75504bc27ceb, 20560106dcaf405bbb288e4e3e8affa6
I20240503 14:12:20.226675 23326 raft_consensus.cc:2726] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:20.249140 23419 raft_consensus.cc:482] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.249325 23419 raft_consensus.cc:504] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.250042 23419 leader_election.cc:288] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:20.250787 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:20.251029 21192 raft_consensus.cc:2454] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:20.251493 21386 leader_election.cc:302] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:20.251775 23419 raft_consensus.cc:2781] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 14:12:20.251796 21386 leader_election.cc:334] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.251902 23419 raft_consensus.cc:482] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.251997 23419 raft_consensus.cc:3037] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:20.254110 23249 raft_consensus.cc:1050] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: attempting to promote NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 to VOTER
I20240503 14:12:20.254788 23419 raft_consensus.cc:504] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.255045 23249 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:20.255453 23419 leader_election.cc:288] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 election: Requested vote from peers 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:20.256354 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8675b2b41d0d4058b286e73a43751a48" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:20.256585 21192 raft_consensus.cc:3037] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:20.257725 22834 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:20.258703 23249 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:20.259279 21256 consensus_peers.cc:485] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:20.259356 21192 raft_consensus.cc:2454] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 2.
I20240503 14:12:20.260046 21386 leader_election.cc:302] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
W20240503 14:12:20.260318 21386 leader_election.cc:334] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.260443 23419 raft_consensus.cc:2781] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:20.260699 23419 raft_consensus.cc:686] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:20.260962 23419 consensus_queue.cc:237] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.264314 20918 catalog_manager.cc:5484] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 1 to 2, leader changed from 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 2 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.271015 21192 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:20.271757 23431 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:20.274667 23249 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:20.275357 21192 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:20.278999 22834 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:20.279660 20918 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.284832 21321 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:20.286756 22834 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:20.286823 21192 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:20.287413 23249 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:20.287706 23374 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:20.289799 23431 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:20.290443 21192 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:20.290519 22834 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:20.293951 20905 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 943f587dd47e486f9be763a1c52bf43f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:20.294723 20918 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
W20240503 14:12:20.303977 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:20.305423 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 943f587dd47e486f9be763a1c52bf43f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.312295 23371 raft_consensus.cc:482] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.312466 23371 raft_consensus.cc:504] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:20.313118 23371 leader_election.cc:288] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:20.313477 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:20.313709 21192 raft_consensus.cc:2454] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 1.
I20240503 14:12:20.314499 21256 leader_election.cc:302] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82, bce0dba863d04bcc8448b0d9f3ec710b; no voters:
W20240503 14:12:20.314776 21256 leader_election.cc:334] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.314826 23371 raft_consensus.cc:2781] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:20.314940 23371 raft_consensus.cc:482] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.315022 23371 raft_consensus.cc:3037] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:20.317808 23371 raft_consensus.cc:504] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:20.318392 23371 leader_election.cc:288] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 2 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:20.318866 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7258a94fee4aa49a19fe1436526abd" candidate_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:20.319073 21192 raft_consensus.cc:3037] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:12:20.319674 21256 leader_election.cc:334] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.321914 21192 raft_consensus.cc:2454] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bce0dba863d04bcc8448b0d9f3ec710b in term 2.
I20240503 14:12:20.322365 21256 leader_election.cc:302] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82, bce0dba863d04bcc8448b0d9f3ec710b; no voters: 1114934d9a3544a9935d75504bc27ceb
I20240503 14:12:20.322659 23371 raft_consensus.cc:2781] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:20.322875 23371 raft_consensus.cc:686] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [term 2 LEADER]: Becoming Leader. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Running, Role: LEADER
I20240503 14:12:20.323269 23371 consensus_queue.cc:237] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:20.326486 20911 catalog_manager.cc:5484] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: term changed from 1 to 2, leader changed from 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) to bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3). New cstate: current_term: 2 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.448822 23419 raft_consensus.cc:482] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.449049 23419 raft_consensus.cc:504] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.449715 23419 leader_election.cc:288] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:20.450311 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82" is_pre_election: true
I20240503 14:12:20.450579 21192 raft_consensus.cc:2454] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 1.
I20240503 14:12:20.451278 21386 leader_election.cc:302] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters:
W20240503 14:12:20.451514 21386 leader_election.cc:334] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.451571 23419 raft_consensus.cc:2781] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:20.451671 23419 raft_consensus.cc:482] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1114934d9a3544a9935d75504bc27ceb)
I20240503 14:12:20.451768 23419 raft_consensus.cc:3037] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:20.453720 23419 raft_consensus.cc:504] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.454181 23419 leader_election.cc:288] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:12:20.454679 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6b3d66deae41439fcbf13cb5400045" candidate_uuid: "20560106dcaf405bbb288e4e3e8affa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:20.454876 21192 raft_consensus.cc:3037] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:12:20.455503 21386 leader_election.cc:334] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:20.457144 21192 raft_consensus.cc:2454] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20560106dcaf405bbb288e4e3e8affa6 in term 2.
I20240503 14:12:20.457520 21386 leader_election.cc:302] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82; no voters: 1114934d9a3544a9935d75504bc27ceb
I20240503 14:12:20.457854 23419 raft_consensus.cc:2781] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:20.458015 23419 raft_consensus.cc:686] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 LEADER]: Becoming Leader. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:20.458264 23419 consensus_queue.cc:237] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.460878 20911 catalog_manager.cc:5484] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: term changed from 1 to 2, leader changed from 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) to 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4). New cstate: current_term: 2 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.586354 21452 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:12:20.588670 21321 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:20.589430 23418 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:20.592200 23418 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } }
W20240503 14:12:20.592877 21386 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:20.593082 21321 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } }
I20240503 14:12:20.597143 20911 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:20.602720 21452 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:20.605175 21321 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:20.607479 23418 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:20.610764 23419 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:20.611454 21321 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:20.616063 20906 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:12:20.617291 20911 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 4 to 5, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
W20240503 14:12:20.620927 21386 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:20.621030 21385 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925): Couldn't send request to peer eb49440048bd4d749779867fa4ae1489. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e999dc5b3bb441ba372e97ac0d6dcb0. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:20.758630 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:20.773526 21192 raft_consensus.cc:1261] T de7258a94fee4aa49a19fe1436526abd P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:20.774142 23371 consensus_queue.cc:1035] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:12:20.779692 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:20.865667 21192 raft_consensus.cc:1261] T 8675b2b41d0d4058b286e73a43751a48 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:20.866329 23462 consensus_queue.cc:1035] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:20.944362 21191 raft_consensus.cc:1261] T bd6b3d66deae41439fcbf13cb5400045 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:12:20.944909 23419 consensus_queue.cc:1035] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:12:20.949383 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:21.143011 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:21.219409 23481 ts_tablet_manager.cc:902] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:21.221141 23481 tablet_copy_client.cc:308] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:21.227996 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 from peer eb49440048bd4d749779867fa4ae1489 ({username='slave'} at 127.20.99.5:37943)
I20240503 14:12:21.228251 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 from peer eb49440048bd4d749779867fa4ae1489 at {username='slave'} at 127.20.99.5:37943: session id = eb49440048bd4d749779867fa4ae1489-0e999dc5b3bb441ba372e97ac0d6dcb0
I20240503 14:12:21.230792 21472 tablet_copy_source_session.cc:205] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:21.235548 23481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.248546 23481 tablet_copy_client.cc:783] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:21.248865 23481 tablet_copy_client.cc:647] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:21.251020 23485 tablet_copy_client.cc:1002] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:21.519145 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 943f587dd47e486f9be763a1c52bf43f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:21.603626 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:21.784754 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:21.962692 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:22.071900 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:22.186542 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:22.587796 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 943f587dd47e486f9be763a1c52bf43f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:22.751566 23481 tablet_copy_client.cc:523] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:22.755054 23481 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Bootstrap starting.
I20240503 14:12:22.764283 23481 log.cc:824] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:22.796738 23481 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:22.797250 23481 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Bootstrap complete.
I20240503 14:12:22.797675 23481 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.004s
I20240503 14:12:22.802173 23481 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:22.802608 23481 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Initialized, Role: LEARNER
I20240503 14:12:22.803112 23481 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:22.804734 23481 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 14:12:22.805902 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session eb49440048bd4d749779867fa4ae1489-0e999dc5b3bb441ba372e97ac0d6dcb0 received from {username='slave'} at 127.20.99.5:37943
I20240503 14:12:22.806054 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session eb49440048bd4d749779867fa4ae1489-0e999dc5b3bb441ba372e97ac0d6dcb0 on tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with peer eb49440048bd4d749779867fa4ae1489
I20240503 14:12:22.912551 22571 raft_consensus.cc:1203] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:12:23.043514 21386 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:23.311699 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:23.332568 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:12:23.364794 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:23.501869 23551 raft_consensus.cc:1050] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER eb49440048bd4d749779867fa4ae1489 to VOTER
I20240503 14:12:23.502856 23551 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:23.505609 22571 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:23.506448 21321 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:23.506320 23553 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:23.506640 21386 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.507009 23553 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:23.510033 23551 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:23.510746 22571 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:23.513760 21321 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:23.514633 20915 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 5 to 6, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:23.520797 21452 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:23.522632 21321 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:23.523198 23419 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:23.523301 22571 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:23.523932 23419 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:23.525434 23545 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:23.526017 21321 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:23.528549 20906 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:23.529875 22571 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:23.530368 20918 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
W20240503 14:12:23.621265 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:23.757933 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:12:23.778291 21321 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:12:23.780640 21191 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:12:23.781652 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.781716 23575 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:23.784341 23374 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } }
I20240503 14:12:23.785001 21191 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } }
I20240503 14:12:23.790982 20915 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } }
I20240503 14:12:23.802109 21321 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:23.817750 21191 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:23.819166 23577 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:12:23.819540 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:23.819830 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487): Couldn't send request to peer bb8d27ca78834bacb9ef4a67a829b494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98f8323a5a1c45ec97a4ccbc937fb135. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.821873 23374 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:23.822516 21191 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:23.826515 20905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98f8323a5a1c45ec97a4ccbc937fb135 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:12:23.828963 20915 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 4 to 5, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:24.000082 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:12:24.183574 23591 ts_tablet_manager.cc:902] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Initiating tablet copy from peer bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:24.185412 23591 tablet_copy_client.cc:308] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43159
W20240503 14:12:24.189625 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:12:24.190615 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:12:24.193089 21342 tablet_copy_service.cc:140] P bce0dba863d04bcc8448b0d9f3ec710b: Received BeginTabletCopySession request for tablet 98f8323a5a1c45ec97a4ccbc937fb135 from peer bb8d27ca78834bacb9ef4a67a829b494 ({username='slave'} at 127.20.99.8:51385)
I20240503 14:12:24.193274 21342 tablet_copy_service.cc:161] P bce0dba863d04bcc8448b0d9f3ec710b: Beginning new tablet copy session on tablet 98f8323a5a1c45ec97a4ccbc937fb135 from peer bb8d27ca78834bacb9ef4a67a829b494 at {username='slave'} at 127.20.99.8:51385: session id = bb8d27ca78834bacb9ef4a67a829b494-98f8323a5a1c45ec97a4ccbc937fb135
I20240503 14:12:24.195137 21342 tablet_copy_source_session.cc:205] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:24.199398 23591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f8323a5a1c45ec97a4ccbc937fb135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.209739 23591 tablet_copy_client.cc:783] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:24.210064 23591 tablet_copy_client.cc:647] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:24.212024 23595 tablet_copy_client.cc:1002] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:24.220254 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 14:12:24.334698 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:12:24.367936 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 14:12:24.679651 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:24.992991 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:12:25.712651 23591 tablet_copy_client.cc:523] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:25.717471 23591 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap starting.
I20240503 14:12:25.727743 23591 log.cc:824] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Log is configured to *not* fsync() on all Append() calls
W20240503 14:12:25.731994 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 14:12:25.733245 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:25.761204 23591 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:25.761688 23591 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap complete.
I20240503 14:12:25.762109 23591 ts_tablet_manager.cc:1367] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Time spent bootstrapping tablet: real 0.045s user 0.032s sys 0.012s
I20240503 14:12:25.767680 23591 raft_consensus.cc:348] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:25.768815 23591 raft_consensus.cc:729] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb8d27ca78834bacb9ef4a67a829b494, State: Initialized, Role: LEARNER
I20240503 14:12:25.770105 23591 consensus_queue.cc:260] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:25.772033 23591 ts_tablet_manager.cc:1398] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494: Time spent starting tablet: real 0.010s user 0.007s sys 0.005s
I20240503 14:12:25.772818 21342 tablet_copy_service.cc:342] P bce0dba863d04bcc8448b0d9f3ec710b: Request end of tablet copy session bb8d27ca78834bacb9ef4a67a829b494-98f8323a5a1c45ec97a4ccbc937fb135 received from {username='slave'} at 127.20.99.8:51385
I20240503 14:12:25.772927 21342 tablet_copy_service.cc:433] P bce0dba863d04bcc8448b0d9f3ec710b: ending tablet copy session bb8d27ca78834bacb9ef4a67a829b494-98f8323a5a1c45ec97a4ccbc937fb135 on tablet 98f8323a5a1c45ec97a4ccbc937fb135 with peer bb8d27ca78834bacb9ef4a67a829b494
W20240503 14:12:25.789840 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:12:25.802935 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:12:25.981104 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:12:26.107017 22965 raft_consensus.cc:1203] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:12:26.162657 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:26.348687 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:26.357890 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:26.599349 23669 raft_consensus.cc:1050] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: attempting to promote NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 to VOTER
I20240503 14:12:26.600207 23669 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:26.602978 21191 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:26.603579 23608 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:26.603912 22965 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:26.604712 23608 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:26.604930 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:26.607957 23671 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:26.608731 22965 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:26.611567 21191 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:26.612803 20918 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 5 to 6, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:26.623471 21321 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:26.625334 22965 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:26.625384 21191 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:26.626020 23671 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:26.626494 23671 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:26.628157 23671 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:26.628836 21191 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:26.629616 22965 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:26.631749 20905 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98f8323a5a1c45ec97a4ccbc937fb135 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:26.633617 20915 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
W20240503 14:12:26.677974 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:26.751379 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98f8323a5a1c45ec97a4ccbc937fb135: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:26.833034 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:12:26.994820 21191 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:26.997706 21452 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:26.998308 23693 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:12:26.998739 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.000926 23326 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } }
I20240503 14:12:27.001485 21452 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } }
I20240503 14:12:27.006132 20915 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:27.011116 21191 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:27.015048 21452 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:12:27.015861 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.016036 23695 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:27.019572 23326 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:27.020318 21452 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
W20240503 14:12:27.024108 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299): Couldn't send request to peer 748b2b25b1b94a1b94788e8ece4ecb6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b50c6f09335d4cb4a2c2288303fc1a7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.024578 20904 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b50c6f09335d4cb4a2c2288303fc1a7f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:12:27.025889 20918 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 4 to 5, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:27.486040 23713 ts_tablet_manager.cc:902] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Initiating tablet copy from peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:27.487881 23713 tablet_copy_client.cc:308] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:33731
I20240503 14:12:27.494045 21212 tablet_copy_service.cc:140] P 8cc13f87df2343dcb1c50aae26230e82: Received BeginTabletCopySession request for tablet b50c6f09335d4cb4a2c2288303fc1a7f from peer 748b2b25b1b94a1b94788e8ece4ecb6d ({username='slave'} at 127.20.99.6:35819)
I20240503 14:12:27.494294 21212 tablet_copy_service.cc:161] P 8cc13f87df2343dcb1c50aae26230e82: Beginning new tablet copy session on tablet b50c6f09335d4cb4a2c2288303fc1a7f from peer 748b2b25b1b94a1b94788e8ece4ecb6d at {username='slave'} at 127.20.99.6:35819: session id = 748b2b25b1b94a1b94788e8ece4ecb6d-b50c6f09335d4cb4a2c2288303fc1a7f
I20240503 14:12:27.497023 21212 tablet_copy_source_session.cc:205] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:27.501791 23713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50c6f09335d4cb4a2c2288303fc1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.511615 23713 tablet_copy_client.cc:783] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:27.511936 23713 tablet_copy_client.cc:647] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:27.513944 23717 tablet_copy_client.cc:1002] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:27.636581 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:27.796092 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98f8323a5a1c45ec97a4ccbc937fb135: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:28.123518 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:28.446517 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:28.460533 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:12:28.494279 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:12:28.735273 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:12:28.826668 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 943f587dd47e486f9be763a1c52bf43f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:29.014549 23713 tablet_copy_client.cc:523] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.018458 23713 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap starting.
I20240503 14:12:29.027595 23713 log.cc:824] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:29.059976 23713 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.060478 23713 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap complete.
I20240503 14:12:29.060890 23713 ts_tablet_manager.cc:1367] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.005s
I20240503 14:12:29.065258 23713 raft_consensus.cc:348] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:29.065654 23713 raft_consensus.cc:729] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 748b2b25b1b94a1b94788e8ece4ecb6d, State: Initialized, Role: LEARNER
I20240503 14:12:29.066129 23713 consensus_queue.cc:260] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:29.067494 23713 ts_tablet_manager.cc:1398] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240503 14:12:29.068470 21212 tablet_copy_service.cc:342] P 8cc13f87df2343dcb1c50aae26230e82: Request end of tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-b50c6f09335d4cb4a2c2288303fc1a7f received from {username='slave'} at 127.20.99.6:35819
I20240503 14:12:29.068616 21212 tablet_copy_service.cc:433] P 8cc13f87df2343dcb1c50aae26230e82: ending tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-b50c6f09335d4cb4a2c2288303fc1a7f on tablet b50c6f09335d4cb4a2c2288303fc1a7f with peer 748b2b25b1b94a1b94788e8ece4ecb6d
I20240503 14:12:29.092412 22703 raft_consensus.cc:1203] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:12:29.123893 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:12:29.381876 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:12:29.589432 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:12:29.632761 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 26 similar messages]
I20240503 14:12:29.651767 23784 raft_consensus.cc:1050] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: attempting to promote NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d to VOTER
I20240503 14:12:29.652868 23784 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:29.655716 22703 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:29.656388 23747 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:29.656634 21452 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:29.657332 23747 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:29.657611 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.668562 23784 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:29.669566 22703 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:29.673574 21452 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:29.675170 20918 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 5 to 6, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.682188 21191 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:29.683964 22703 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:29.684057 21452 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:29.684517 23788 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:29.685113 23747 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:29.687685 23783 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:29.688295 22703 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:29.688371 21452 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:29.692234 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b50c6f09335d4cb4a2c2288303fc1a7f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:29.692813 20911 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d reported cstate change: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
W20240503 14:12:29.898561 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b50c6f09335d4cb4a2c2288303fc1a7f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:29.970803 21191 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:12:29.972795 21452 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:29.973356 23783 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:12:29.973635 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.976238 23788 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } }
I20240503 14:12:29.976938 21452 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } }
I20240503 14:12:29.980929 20918 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:29.985692 21191 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:29.990417 21452 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:29.991717 23788 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:29.998483 23747 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
W20240503 14:12:30.000851 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.001607 21451 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:30.007565 20904 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet df4f572b770c4fdf8d91d4f449f579eb with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:12:30.008191 20915 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 4 to 5, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
W20240503 14:12:30.014263 21125 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693): Couldn't send request to peer ad5e6850bbff47fd9ff2e14de5b48223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df4f572b770c4fdf8d91d4f449f579eb. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:12:30.187717 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:12:30.348186 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:12:30.457769 23827 ts_tablet_manager.cc:902] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: Initiating tablet copy from peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:30.459041 23827 tablet_copy_client.cc:308] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:33731
I20240503 14:12:30.464802 21212 tablet_copy_service.cc:140] P 8cc13f87df2343dcb1c50aae26230e82: Received BeginTabletCopySession request for tablet df4f572b770c4fdf8d91d4f449f579eb from peer ad5e6850bbff47fd9ff2e14de5b48223 ({username='slave'} at 127.20.99.7:53587)
I20240503 14:12:30.464991 21212 tablet_copy_service.cc:161] P 8cc13f87df2343dcb1c50aae26230e82: Beginning new tablet copy session on tablet df4f572b770c4fdf8d91d4f449f579eb from peer ad5e6850bbff47fd9ff2e14de5b48223 at {username='slave'} at 127.20.99.7:53587: session id = ad5e6850bbff47fd9ff2e14de5b48223-df4f572b770c4fdf8d91d4f449f579eb
I20240503 14:12:30.466917 21212 tablet_copy_source_session.cc:205] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:30.468107 23827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4f572b770c4fdf8d91d4f449f579eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.472468 23827 tablet_copy_client.cc:783] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:30.472762 23827 tablet_copy_client.cc:647] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:30.474376 23831 tablet_copy_client.cc:1002] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:30.799556 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:12:30.832499 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240503 14:12:30.881700 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:12:30.927804 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98f8323a5a1c45ec97a4ccbc937fb135: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:30.944983 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:12:31.477743 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:12:31.938530 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:31.974848 23827 tablet_copy_client.cc:523] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:31.977757 23827 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap starting.
I20240503 14:12:32.008620 23827 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:32.009037 23827 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap complete.
I20240503 14:12:32.009297 23827 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.007s
I20240503 14:12:32.010097 23827 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:32.010309 23827 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad5e6850bbff47fd9ff2e14de5b48223, State: Initialized, Role: LEARNER
I20240503 14:12:32.010457 23827 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:32.011061 23827 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:12:32.011919 21212 tablet_copy_service.cc:342] P 8cc13f87df2343dcb1c50aae26230e82: Request end of tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-df4f572b770c4fdf8d91d4f449f579eb received from {username='slave'} at 127.20.99.7:53587
I20240503 14:12:32.012059 21212 tablet_copy_service.cc:433] P 8cc13f87df2343dcb1c50aae26230e82: ending tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-df4f572b770c4fdf8d91d4f449f579eb on tablet df4f572b770c4fdf8d91d4f449f579eb with peer ad5e6850bbff47fd9ff2e14de5b48223
I20240503 14:12:32.231963 22834 raft_consensus.cc:1203] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:12:32.485670 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:12:32.774610 23900 raft_consensus.cc:1050] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: attempting to promote NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 to VOTER
I20240503 14:12:32.775597 23900 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:32.778465 22834 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:32.778517 21451 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:32.779139 23903 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:32.779503 23900 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:32.779877 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:32.782501 23903 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:32.783114 22834 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:32.786286 21451 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:32.787034 20918 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 5 to 6, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:12:32.796301 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:12:32.799928 21191 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:32.801716 22834 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:32.801785 21451 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:32.802311 23900 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:32.802592 23901 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:32.804893 23903 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:32.805567 21451 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:32.805568 22834 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:32.809069 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df4f572b770c4fdf8d91d4f449f579eb with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:32.810768 20911 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 6 to 7, VOTER 1114934d9a3544a9935d75504bc27ceb (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:12:33.015969 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df4f572b770c4fdf8d91d4f449f579eb: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:33.049855 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:12:33.192649 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:12:33.248533 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:12:33.277537 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.286412 21431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:33.295075 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.313280 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.319555 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.335026 21302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:33.359869 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:12:33.393707 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:12:33.527654 21431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:33.546228 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.546458 21302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:33.546905 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.549019 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.557667 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.565418 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.670531 21451 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:33.673048 21320 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:33.673787 21191 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:33.674530 23838 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:33.675276 23962 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:12:33.683081 21386 consensus_peers.cc:485] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487): Couldn't send request to peer bb8d27ca78834bacb9ef4a67a829b494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8c3db38486c4d41a85551928c0f2202. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.687265 23838 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:33.688118 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:33.689882 21320 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:33.693475 20915 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index -1 to 4, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:33.708498 21451 consensus_queue.cc:237] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:33.710803 21320 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:33.710918 21192 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:33.711513 24001 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:33.711748 23962 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:12:33.712025 21385 consensus_peers.cc:485] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925): Couldn't send request to peer eb49440048bd4d749779867fa4ae1489. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f2c7555d83d4300b8b99e186e92a794. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:33.714350 23962 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:33.714903 21192 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:33.715375 21320 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:33.719120 20918 catalog_manager.cc:5484] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:33.911509 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:12:34.098999 24037 ts_tablet_manager.cc:902] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:34.100921 24037 tablet_copy_client.cc:308] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:34.107221 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet b8c3db38486c4d41a85551928c0f2202 from peer bb8d27ca78834bacb9ef4a67a829b494 ({username='slave'} at 127.20.99.8:39557)
I20240503 14:12:34.107442 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet b8c3db38486c4d41a85551928c0f2202 from peer bb8d27ca78834bacb9ef4a67a829b494 at {username='slave'} at 127.20.99.8:39557: session id = bb8d27ca78834bacb9ef4a67a829b494-b8c3db38486c4d41a85551928c0f2202
I20240503 14:12:34.109405 21472 tablet_copy_source_session.cc:205] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:34.110677 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c3db38486c4d41a85551928c0f2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:34.115314 24037 tablet_copy_client.cc:783] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:34.115563 24037 tablet_copy_client.cc:647] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:34.117164 24041 tablet_copy_client.cc:1002] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Injecting latency into file download: 1500
I20240503 14:12:34.257864 24053 ts_tablet_manager.cc:902] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:34.259243 24053 tablet_copy_client.cc:308] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:34.260001 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet 2f2c7555d83d4300b8b99e186e92a794 from peer eb49440048bd4d749779867fa4ae1489 ({username='slave'} at 127.20.99.5:37943)
I20240503 14:12:34.260192 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet 2f2c7555d83d4300b8b99e186e92a794 from peer eb49440048bd4d749779867fa4ae1489 at {username='slave'} at 127.20.99.5:37943: session id = eb49440048bd4d749779867fa4ae1489-2f2c7555d83d4300b8b99e186e92a794
I20240503 14:12:34.263337 21472 tablet_copy_source_session.cc:205] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:34.264658 24053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f2c7555d83d4300b8b99e186e92a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:34.271205 24053 tablet_copy_client.cc:783] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:34.271512 24053 tablet_copy_client.cc:647] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:34.273264 24057 tablet_copy_client.cc:1002] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:34.583155 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:12:34.786600 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:12:35.012863 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:12:35.043954 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98f8323a5a1c45ec97a4ccbc937fb135: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:35.313710 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:12:35.617584 24037 tablet_copy_client.cc:523] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:35.620539 24037 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap starting.
I20240503 14:12:35.655936 24037 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:35.656548 24037 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap complete.
I20240503 14:12:35.656912 24037 ts_tablet_manager.cc:1367] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: Time spent bootstrapping tablet: real 0.036s user 0.037s sys 0.000s
I20240503 14:12:35.658022 24037 raft_consensus.cc:348] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:35.658371 24037 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb8d27ca78834bacb9ef4a67a829b494, State: Initialized, Role: LEARNER
I20240503 14:12:35.658607 24037 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:35.659530 24037 ts_tablet_manager.cc:1398] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:35.660305 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session bb8d27ca78834bacb9ef4a67a829b494-b8c3db38486c4d41a85551928c0f2202 received from {username='slave'} at 127.20.99.8:39557
I20240503 14:12:35.660468 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session bb8d27ca78834bacb9ef4a67a829b494-b8c3db38486c4d41a85551928c0f2202 on tablet b8c3db38486c4d41a85551928c0f2202 with peer bb8d27ca78834bacb9ef4a67a829b494
W20240503 14:12:35.682533 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:12:35.717206 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:12:35.757484 22965 raft_consensus.cc:1203] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:35.773816 24053 tablet_copy_client.cc:523] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:35.776970 24053 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: Bootstrap starting.
W20240503 14:12:35.794273 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:12:35.809465 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:12:35.810074 24053 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:35.810461 24053 tablet_bootstrap.cc:492] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: Bootstrap complete.
I20240503 14:12:35.810688 24053 ts_tablet_manager.cc:1367] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.004s
I20240503 14:12:35.811550 24053 raft_consensus.cc:348] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:35.811764 24053 raft_consensus.cc:729] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Initialized, Role: LEARNER
I20240503 14:12:35.811918 24053 consensus_queue.cc:260] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:35.812600 24053 ts_tablet_manager.cc:1398] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:12:35.813434 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session eb49440048bd4d749779867fa4ae1489-2f2c7555d83d4300b8b99e186e92a794 received from {username='slave'} at 127.20.99.5:37943
I20240503 14:12:35.813611 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session eb49440048bd4d749779867fa4ae1489-2f2c7555d83d4300b8b99e186e92a794 on tablet 2f2c7555d83d4300b8b99e186e92a794 with peer eb49440048bd4d749779867fa4ae1489
I20240503 14:12:35.840698 22571 raft_consensus.cc:1203] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:36.224524 24081 raft_consensus.cc:1050] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 to VOTER
I20240503 14:12:36.225306 24081 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:36.228117 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:36.228092 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:36.228400 21320 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:36.228793 24160 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.229187 24081 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.229445 24161 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.232682 24161 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.233340 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.233821 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.236682 21320 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.238984 20910 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 4 to 5, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.252166 21452 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:36.253960 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:36.253938 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:36.254797 24161 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:36.255156 24081 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:36.257596 24160 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.258256 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.258335 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.262341 20906 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8c3db38486c4d41a85551928c0f2202 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:36.263234 20917 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:36.267648 21302 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c3db38486c4d41a85551928c0f2202 with delete_type TABLET_DATA_TOMBSTONED (TS bce0dba863d04bcc8448b0d9f3ec710b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43526
I20240503 14:12:36.271765 24171 tablet_replica.cc:330] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: stopping tablet replica
I20240503 14:12:36.272187 24171 raft_consensus.cc:2229] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:36.272493 24171 raft_consensus.cc:2258] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:36.273905 24171 ts_tablet_manager.cc:1875] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:36.278882 24171 ts_tablet_manager.cc:1888] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:12:36.279031 24171 log.cc:1196] T b8c3db38486c4d41a85551928c0f2202 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8c3db38486c4d41a85551928c0f2202
I20240503 14:12:36.279953 20905 catalog_manager.cc:4830] TS bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159): tablet b8c3db38486c4d41a85551928c0f2202 (table rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]) successfully deleted
I20240503 14:12:36.293999 24163 raft_consensus.cc:1050] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER eb49440048bd4d749779867fa4ae1489 to VOTER
I20240503 14:12:36.294939 24163 consensus_queue.cc:237] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:36.297686 21320 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:36.297696 21192 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:36.298238 22571 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:36.298202 24163 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.298506 24081 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.298769 24161 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.302284 24161 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.302901 22571 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.303565 21192 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.306011 21320 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.308915 20917 catalog_manager.cc:5484] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.314621 21452 consensus_queue.cc:237] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
W20240503 14:12:36.316556 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:12:36.320225 21320 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:36.320873 22571 raft_consensus.cc:1261] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:36.321071 24163 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:36.321640 24163 consensus_queue.cc:1035] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:36.327872 24081 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.330935 20906 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2f2c7555d83d4300b8b99e186e92a794 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:36.332438 20918 catalog_manager.cc:5484] T 2f2c7555d83d4300b8b99e186e92a794 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 5 to 6, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:36.332408 21320 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.332371 22571 raft_consensus.cc:2932] T 2f2c7555d83d4300b8b99e186e92a794 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:36.336741 21171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f2c7555d83d4300b8b99e186e92a794 with delete_type TABLET_DATA_TOMBSTONED (TS 8cc13f87df2343dcb1c50aae26230e82 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50222
I20240503 14:12:36.339670 24176 tablet_replica.cc:330] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: stopping tablet replica
I20240503 14:12:36.340387 24176 raft_consensus.cc:2229] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:36.340863 24176 raft_consensus.cc:2258] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:36.342502 24176 ts_tablet_manager.cc:1875] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:36.349057 24176 ts_tablet_manager.cc:1888] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:12:36.349191 24176 log.cc:1196] T 2f2c7555d83d4300b8b99e186e92a794 P 8cc13f87df2343dcb1c50aae26230e82: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f2c7555d83d4300b8b99e186e92a794
I20240503 14:12:36.350006 20904 catalog_manager.cc:4830] TS 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): tablet 2f2c7555d83d4300b8b99e186e92a794 (table rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]) successfully deleted
I20240503 14:12:36.388381 21320 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:36.392153 22834 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:36.392710 21192 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:36.394687 23858 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240503 14:12:36.395376 24194 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:36.397668 24194 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:36.398352 22834 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:36.399148 21192 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
W20240503 14:12:36.403692 21256 consensus_peers.cc:485] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299): Couldn't send request to peer 748b2b25b1b94a1b94788e8ece4ecb6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 943f587dd47e486f9be763a1c52bf43f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:36.403982 20915 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 reported cstate change: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:36.417546 21192 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:36.420567 21452 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:36.421151 22834 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:36.421272 24207 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:36.421838 23900 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:36.424710 24207 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:36.425448 21452 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:36.430423 20910 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 7 to 8, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:36.430848 21125 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925): Couldn't send request to peer eb49440048bd4d749779867fa4ae1489. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df4f572b770c4fdf8d91d4f449f579eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:36.435081 22834 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:36.445559 21452 consensus_queue.cc:237] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:36.447824 21192 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:36.447921 21320 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:36.448340 24161 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:36.448678 24160 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:36.451702 24081 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:36.452353 21320 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:36.452353 21192 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:36.457700 20918 catalog_manager.cc:5484] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index -1 to 4, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
W20240503 14:12:36.475267 21385 consensus_peers.cc:485] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693): Couldn't send request to peer ad5e6850bbff47fd9ff2e14de5b48223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a955820c3a79415c9d118901eb40f172. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:36.888176 24251 ts_tablet_manager.cc:902] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:36.889664 24251 tablet_copy_client.cc:308] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:36.897102 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet a955820c3a79415c9d118901eb40f172 from peer ad5e6850bbff47fd9ff2e14de5b48223 ({username='slave'} at 127.20.99.7:41475)
I20240503 14:12:36.897292 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet a955820c3a79415c9d118901eb40f172 from peer ad5e6850bbff47fd9ff2e14de5b48223 at {username='slave'} at 127.20.99.7:41475: session id = ad5e6850bbff47fd9ff2e14de5b48223-a955820c3a79415c9d118901eb40f172
I20240503 14:12:36.899641 21472 tablet_copy_source_session.cc:205] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:36.900883 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a955820c3a79415c9d118901eb40f172. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:36.905390 24251 tablet_copy_client.cc:783] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:36.905669 24251 tablet_copy_client.cc:647] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:36.907474 24255 tablet_copy_client.cc:1002] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Injecting latency into file download: 1500
I20240503 14:12:36.910046 24257 ts_tablet_manager.cc:902] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: Initiating tablet copy from peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:36.911414 24257 tablet_copy_client.cc:308] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:33731
I20240503 14:12:36.917658 21212 tablet_copy_service.cc:140] P 8cc13f87df2343dcb1c50aae26230e82: Received BeginTabletCopySession request for tablet df4f572b770c4fdf8d91d4f449f579eb from peer eb49440048bd4d749779867fa4ae1489 ({username='slave'} at 127.20.99.5:34265)
I20240503 14:12:36.917816 21212 tablet_copy_service.cc:161] P 8cc13f87df2343dcb1c50aae26230e82: Beginning new tablet copy session on tablet df4f572b770c4fdf8d91d4f449f579eb from peer eb49440048bd4d749779867fa4ae1489 at {username='slave'} at 127.20.99.5:34265: session id = eb49440048bd4d749779867fa4ae1489-df4f572b770c4fdf8d91d4f449f579eb
I20240503 14:12:36.919852 21212 tablet_copy_source_session.cc:205] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:36.921185 24257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4f572b770c4fdf8d91d4f449f579eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:36.925952 24257 tablet_copy_client.cc:783] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:36.926200 24257 tablet_copy_client.cc:647] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:36.927932 24261 tablet_copy_client.cc:1002] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: tablet copy: Injecting latency into file download: 1500
I20240503 14:12:36.970247 24278 ts_tablet_manager.cc:902] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: Initiating tablet copy from peer bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:36.971561 24278 tablet_copy_client.cc:308] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43159
I20240503 14:12:36.977566 21342 tablet_copy_service.cc:140] P bce0dba863d04bcc8448b0d9f3ec710b: Received BeginTabletCopySession request for tablet 943f587dd47e486f9be763a1c52bf43f from peer 748b2b25b1b94a1b94788e8ece4ecb6d ({username='slave'} at 127.20.99.6:37075)
I20240503 14:12:36.977722 21342 tablet_copy_service.cc:161] P bce0dba863d04bcc8448b0d9f3ec710b: Beginning new tablet copy session on tablet 943f587dd47e486f9be763a1c52bf43f from peer 748b2b25b1b94a1b94788e8ece4ecb6d at {username='slave'} at 127.20.99.6:37075: session id = 748b2b25b1b94a1b94788e8ece4ecb6d-943f587dd47e486f9be763a1c52bf43f
I20240503 14:12:36.979743 21342 tablet_copy_source_session.cc:205] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:36.981241 24278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943f587dd47e486f9be763a1c52bf43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:36.985812 24278 tablet_copy_client.cc:783] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:36.986071 24278 tablet_copy_client.cc:647] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:36.987846 24282 tablet_copy_client.cc:1002] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:37.032469 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240503 14:12:37.033695 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 943f587dd47e486f9be763a1c52bf43f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:37.854998 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:12:38.107235 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b50c6f09335d4cb4a2c2288303fc1a7f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:38.132000 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:12:38.198423 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:12:38.385167 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:12:38.407835 24251 tablet_copy_client.cc:523] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:38.410621 24251 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap starting.
I20240503 14:12:38.428336 24257 tablet_copy_client.cc:523] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:38.430868 24257 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: Bootstrap starting.
I20240503 14:12:38.442499 24251 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:38.442939 24251 tablet_bootstrap.cc:492] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap complete.
I20240503 14:12:38.443280 24251 ts_tablet_manager.cc:1367] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: Time spent bootstrapping tablet: real 0.033s user 0.035s sys 0.000s
I20240503 14:12:38.444161 24251 raft_consensus.cc:348] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:38.444377 24251 raft_consensus.cc:729] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad5e6850bbff47fd9ff2e14de5b48223, State: Initialized, Role: LEARNER
I20240503 14:12:38.444631 24251 consensus_queue.cc:260] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:38.445430 24251 ts_tablet_manager.cc:1398] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:38.446185 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-a955820c3a79415c9d118901eb40f172 received from {username='slave'} at 127.20.99.7:41475
I20240503 14:12:38.446316 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-a955820c3a79415c9d118901eb40f172 on tablet a955820c3a79415c9d118901eb40f172 with peer ad5e6850bbff47fd9ff2e14de5b48223
I20240503 14:12:38.466570 24257 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:38.467065 24257 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: Bootstrap complete.
I20240503 14:12:38.467383 24257 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: Time spent bootstrapping tablet: real 0.037s user 0.034s sys 0.004s
I20240503 14:12:38.468017 24257 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:38.468298 24257 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Initialized, Role: LEARNER
I20240503 14:12:38.468495 24257 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:38.469175 24257 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:38.470027 21212 tablet_copy_service.cc:342] P 8cc13f87df2343dcb1c50aae26230e82: Request end of tablet copy session eb49440048bd4d749779867fa4ae1489-df4f572b770c4fdf8d91d4f449f579eb received from {username='slave'} at 127.20.99.5:34265
I20240503 14:12:38.470158 21212 tablet_copy_service.cc:433] P 8cc13f87df2343dcb1c50aae26230e82: ending tablet copy session eb49440048bd4d749779867fa4ae1489-df4f572b770c4fdf8d91d4f449f579eb on tablet df4f572b770c4fdf8d91d4f449f579eb with peer eb49440048bd4d749779867fa4ae1489
I20240503 14:12:38.488278 24278 tablet_copy_client.cc:523] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:38.490932 24278 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap starting.
I20240503 14:12:38.521926 24278 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:38.522397 24278 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap complete.
I20240503 14:12:38.522677 24278 ts_tablet_manager.cc:1367] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20240503 14:12:38.523638 24278 raft_consensus.cc:348] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:38.523916 24278 raft_consensus.cc:729] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 748b2b25b1b94a1b94788e8ece4ecb6d, State: Initialized, Role: LEARNER
I20240503 14:12:38.524109 24278 consensus_queue.cc:260] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:38.524866 24278 ts_tablet_manager.cc:1398] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:12:38.525768 21342 tablet_copy_service.cc:342] P bce0dba863d04bcc8448b0d9f3ec710b: Request end of tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-943f587dd47e486f9be763a1c52bf43f received from {username='slave'} at 127.20.99.6:37075
I20240503 14:12:38.525935 21342 tablet_copy_service.cc:433] P bce0dba863d04bcc8448b0d9f3ec710b: ending tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-943f587dd47e486f9be763a1c52bf43f on tablet 943f587dd47e486f9be763a1c52bf43f with peer 748b2b25b1b94a1b94788e8ece4ecb6d
I20240503 14:12:38.566388 22703 raft_consensus.cc:1203] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 14:12:38.628437 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:12:38.808981 22571 raft_consensus.cc:1203] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:12:38.904613 22834 raft_consensus.cc:1203] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:38.952843 24426 raft_consensus.cc:1050] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b: attempting to promote NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d to VOTER
I20240503 14:12:38.953557 24426 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:38.955854 22703 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:38.955991 22834 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:38.956064 21191 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:38.956478 24426 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:38.956851 24425 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:38.957173 24428 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:38.960391 24428 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.961239 21191 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.961542 22703 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.963861 22834 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.964975 20918 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:38.975270 21320 consensus_queue.cc:237] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:38.977022 22834 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:38.977028 22703 raft_consensus.cc:1261] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:38.977561 24428 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:38.977851 24426 consensus_queue.cc:1035] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:38.980144 24195 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.980790 22703 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.980844 22834 raft_consensus.cc:2932] T 943f587dd47e486f9be763a1c52bf43f P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:38.983301 20905 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 943f587dd47e486f9be763a1c52bf43f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:38.984544 20917 catalog_manager.cc:5484] T 943f587dd47e486f9be763a1c52bf43f P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:38.988327 21171 tablet_service.cc:1514] Processing DeleteTablet for tablet 943f587dd47e486f9be763a1c52bf43f with delete_type TABLET_DATA_TOMBSTONED (TS 8cc13f87df2343dcb1c50aae26230e82 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50222
I20240503 14:12:38.989054 24436 tablet_replica.cc:330] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: stopping tablet replica
I20240503 14:12:38.989434 24436 raft_consensus.cc:2229] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:38.989800 24436 raft_consensus.cc:2258] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:38.990633 24436 ts_tablet_manager.cc:1875] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:38.994853 24436 ts_tablet_manager.cc:1888] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:12:38.994968 24436 log.cc:1196] T 943f587dd47e486f9be763a1c52bf43f P 8cc13f87df2343dcb1c50aae26230e82: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/943f587dd47e486f9be763a1c52bf43f
I20240503 14:12:38.995726 20904 catalog_manager.cc:4830] TS 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): tablet 943f587dd47e486f9be763a1c52bf43f (table rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]) successfully deleted
W20240503 14:12:39.037264 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:12:39.043877 21320 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
W20240503 14:12:39.046487 21256 consensus_peers.cc:485] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299): Couldn't send request to peer 748b2b25b1b94a1b94788e8ece4ecb6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98f8323a5a1c45ec97a4ccbc937fb135. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:39.046484 22965 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:39.046701 21191 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:39.047073 24425 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:39.047463 24195 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:39.050312 24426 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:39.050953 22965 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:39.051067 21191 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:39.055810 20915 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 reported cstate change: config changed from index 7 to 8, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:39.198305 24478 raft_consensus.cc:1050] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: attempting to promote NON_VOTER eb49440048bd4d749779867fa4ae1489 to VOTER
I20240503 14:12:39.199012 24478 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:39.201644 22834 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:39.201658 22571 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:39.201735 21451 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:39.202231 24480 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:39.202641 24305 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:39.202924 24478 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:39.209843 21451 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.209268 24480 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.215090 20917 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 8 to 9, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.218070 22571 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.219566 22834 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.222280 21191 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:39.224531 22833 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:39.225183 22570 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:39.225286 24480 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:39.225770 24477 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:39.228029 24477 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.228621 22833 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.229285 22570 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:39.231400 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df4f572b770c4fdf8d91d4f449f579eb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:39.232458 20910 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:39.236649 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet df4f572b770c4fdf8d91d4f449f579eb with delete_type TABLET_DATA_TOMBSTONED (TS 20560106dcaf405bbb288e4e3e8affa6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35362
I20240503 14:12:39.237529 24484 tablet_replica.cc:330] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: stopping tablet replica
I20240503 14:12:39.237941 24484 raft_consensus.cc:2229] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:39.238400 24484 raft_consensus.cc:2258] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:39.239776 24484 ts_tablet_manager.cc:1875] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:39.244191 24484 ts_tablet_manager.cc:1888] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:12:39.244323 24484 log.cc:1196] T df4f572b770c4fdf8d91d4f449f579eb P 20560106dcaf405bbb288e4e3e8affa6: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/df4f572b770c4fdf8d91d4f449f579eb
I20240503 14:12:39.245108 20906 catalog_manager.cc:4830] TS 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461): tablet df4f572b770c4fdf8d91d4f449f579eb (table rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]) successfully deleted
I20240503 14:12:39.337394 24381 raft_consensus.cc:1050] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 to VOTER
I20240503 14:12:39.338537 24381 consensus_queue.cc:237] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:39.345520 21320 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:39.345566 22833 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:39.346132 24381 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:39.346575 21191 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:39.347769 24381 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:39.348078 24500 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:39.354844 24500 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.356003 22833 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.356443 21191 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.360060 21320 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.361214 20915 catalog_manager.cc:5484] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 4 to 5, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:39.367010 21451 consensus_queue.cc:237] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:39.368907 21192 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:39.369385 24476 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:39.370075 22833 raft_consensus.cc:1261] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:12:39.370565 24476 consensus_queue.cc:1035] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:39.371500 24381 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.372095 21192 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:39.373657 22833 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.375070 22703 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:39.375602 21451 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:39.375669 24480 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:39.376494 21192 raft_consensus.cc:2932] T a955820c3a79415c9d118901eb40f172 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.379657 20917 catalog_manager.cc:5484] T a955820c3a79415c9d118901eb40f172 P ad5e6850bbff47fd9ff2e14de5b48223 reported cstate change: config changed from index 5 to 6, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:39.380635 24480 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:39.385823 21302 tablet_service.cc:1514] Processing DeleteTablet for tablet a955820c3a79415c9d118901eb40f172 with delete_type TABLET_DATA_TOMBSTONED (TS bce0dba863d04bcc8448b0d9f3ec710b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43526
W20240503 14:12:39.386337 21126 consensus_peers.cc:485] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 -> Peer bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487): Couldn't send request to peer bb8d27ca78834bacb9ef4a67a829b494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b50c6f09335d4cb4a2c2288303fc1a7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:39.387830 20906 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a955820c3a79415c9d118901eb40f172 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:39.396085 24514 tablet_replica.cc:330] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: stopping tablet replica
I20240503 14:12:39.396394 24514 raft_consensus.cc:2229] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:39.396881 24514 raft_consensus.cc:2258] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:39.398906 24514 ts_tablet_manager.cc:1875] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:39.398756 24478 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:39.404438 20910 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:39.406236 21451 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:39.408360 24514 ts_tablet_manager.cc:1888] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:12:39.408488 24514 log.cc:1196] T a955820c3a79415c9d118901eb40f172 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/a955820c3a79415c9d118901eb40f172
I20240503 14:12:39.409479 20905 catalog_manager.cc:4830] TS bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159): tablet a955820c3a79415c9d118901eb40f172 (table rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]) successfully deleted
I20240503 14:12:39.409597 22703 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:39.452517 21320 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:39.455114 21451 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:39.455698 21192 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:39.455772 24426 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:39.456388 24426 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:39.459419 24195 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:39.460314 21451 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
W20240503 14:12:39.462637 21255 consensus_peers.cc:485] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925): Couldn't send request to peer eb49440048bd4d749779867fa4ae1489. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39aaa8ba3e0c43078f94e265a961fe97. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:39.464668 20917 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:39.467456 21192 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:39.638790 24554 ts_tablet_manager.cc:902] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: Initiating tablet copy from peer bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:39.640683 24554 tablet_copy_client.cc:308] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43159
I20240503 14:12:39.641464 21342 tablet_copy_service.cc:140] P bce0dba863d04bcc8448b0d9f3ec710b: Received BeginTabletCopySession request for tablet 98f8323a5a1c45ec97a4ccbc937fb135 from peer 748b2b25b1b94a1b94788e8ece4ecb6d ({username='slave'} at 127.20.99.6:37075)
I20240503 14:12:39.641664 21342 tablet_copy_service.cc:161] P bce0dba863d04bcc8448b0d9f3ec710b: Beginning new tablet copy session on tablet 98f8323a5a1c45ec97a4ccbc937fb135 from peer 748b2b25b1b94a1b94788e8ece4ecb6d at {username='slave'} at 127.20.99.6:37075: session id = 748b2b25b1b94a1b94788e8ece4ecb6d-98f8323a5a1c45ec97a4ccbc937fb135
I20240503 14:12:39.643687 21342 tablet_copy_source_session.cc:205] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:39.645013 24554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f8323a5a1c45ec97a4ccbc937fb135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.649600 24554 tablet_copy_client.cc:783] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:39.649847 24554 tablet_copy_client.cc:647] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:39.651430 24558 tablet_copy_client.cc:1002] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:39.660027 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 14:12:39.810351 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240503 14:12:39.855068 24574 ts_tablet_manager.cc:902] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: Initiating tablet copy from peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:39.856509 24574 tablet_copy_client.cc:308] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:33731
I20240503 14:12:39.862601 21212 tablet_copy_service.cc:140] P 8cc13f87df2343dcb1c50aae26230e82: Received BeginTabletCopySession request for tablet b50c6f09335d4cb4a2c2288303fc1a7f from peer bb8d27ca78834bacb9ef4a67a829b494 ({username='slave'} at 127.20.99.8:47311)
I20240503 14:12:39.862769 21212 tablet_copy_service.cc:161] P 8cc13f87df2343dcb1c50aae26230e82: Beginning new tablet copy session on tablet b50c6f09335d4cb4a2c2288303fc1a7f from peer bb8d27ca78834bacb9ef4a67a829b494 at {username='slave'} at 127.20.99.8:47311: session id = bb8d27ca78834bacb9ef4a67a829b494-b50c6f09335d4cb4a2c2288303fc1a7f
I20240503 14:12:39.864688 21212 tablet_copy_source_session.cc:205] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:39.865929 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50c6f09335d4cb4a2c2288303fc1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.870180 24574 tablet_copy_client.cc:783] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:39.870433 24574 tablet_copy_client.cc:647] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:39.871960 24578 tablet_copy_client.cc:1002] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Injecting latency into file download: 1500
I20240503 14:12:39.874948 24580 ts_tablet_manager.cc:902] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: Initiating tablet copy from peer bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159)
I20240503 14:12:39.876515 24580 tablet_copy_client.cc:308] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43159
I20240503 14:12:39.882671 21342 tablet_copy_service.cc:140] P bce0dba863d04bcc8448b0d9f3ec710b: Received BeginTabletCopySession request for tablet 39aaa8ba3e0c43078f94e265a961fe97 from peer eb49440048bd4d749779867fa4ae1489 ({username='slave'} at 127.20.99.5:59839)
I20240503 14:12:39.882827 21342 tablet_copy_service.cc:161] P bce0dba863d04bcc8448b0d9f3ec710b: Beginning new tablet copy session on tablet 39aaa8ba3e0c43078f94e265a961fe97 from peer eb49440048bd4d749779867fa4ae1489 at {username='slave'} at 127.20.99.5:59839: session id = eb49440048bd4d749779867fa4ae1489-39aaa8ba3e0c43078f94e265a961fe97
I20240503 14:12:39.885033 21342 tablet_copy_source_session.cc:205] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:39.886286 24580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39aaa8ba3e0c43078f94e265a961fe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:39.890798 24580 tablet_copy_client.cc:783] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:39.891048 24580 tablet_copy_client.cc:647] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:39.892766 24584 tablet_copy_client.cc:1002] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:40.080813 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:12:40.139228 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:40.397380 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:12:40.398281 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:12:40.772966 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:12:40.940490 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:12:41.151840 24554 tablet_copy_client.cc:523] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:41.154564 24554 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap starting.
W20240503 14:12:41.178741 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:12:41.187358 24554 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:41.187758 24554 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap complete.
I20240503 14:12:41.188016 24554 ts_tablet_manager.cc:1367] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent bootstrapping tablet: real 0.034s user 0.035s sys 0.000s
I20240503 14:12:41.188778 24554 raft_consensus.cc:348] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:41.189013 24554 raft_consensus.cc:729] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 748b2b25b1b94a1b94788e8ece4ecb6d, State: Initialized, Role: LEARNER
I20240503 14:12:41.189239 24554 consensus_queue.cc:260] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:41.189925 24554 ts_tablet_manager.cc:1398] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:41.190778 21342 tablet_copy_service.cc:342] P bce0dba863d04bcc8448b0d9f3ec710b: Request end of tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-98f8323a5a1c45ec97a4ccbc937fb135 received from {username='slave'} at 127.20.99.6:37075
I20240503 14:12:41.190912 21342 tablet_copy_service.cc:433] P bce0dba863d04bcc8448b0d9f3ec710b: ending tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-98f8323a5a1c45ec97a4ccbc937fb135 on tablet 98f8323a5a1c45ec97a4ccbc937fb135 with peer 748b2b25b1b94a1b94788e8ece4ecb6d
W20240503 14:12:41.238332 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df4f572b770c4fdf8d91d4f449f579eb: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:41.372433 24574 tablet_copy_client.cc:523] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:41.375008 24574 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap starting.
I20240503 14:12:41.393213 24580 tablet_copy_client.cc:523] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:41.395795 24580 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: Bootstrap starting.
I20240503 14:12:41.407711 24574 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:41.408098 24574 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap complete.
I20240503 14:12:41.408409 24574 ts_tablet_manager.cc:1367] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: Time spent bootstrapping tablet: real 0.033s user 0.034s sys 0.000s
I20240503 14:12:41.409052 24574 raft_consensus.cc:348] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:41.409281 24574 raft_consensus.cc:729] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb8d27ca78834bacb9ef4a67a829b494, State: Initialized, Role: LEARNER
I20240503 14:12:41.409474 24574 consensus_queue.cc:260] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:41.410271 24574 ts_tablet_manager.cc:1398] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:41.411235 21212 tablet_copy_service.cc:342] P 8cc13f87df2343dcb1c50aae26230e82: Request end of tablet copy session bb8d27ca78834bacb9ef4a67a829b494-b50c6f09335d4cb4a2c2288303fc1a7f received from {username='slave'} at 127.20.99.8:47311
I20240503 14:12:41.411374 21212 tablet_copy_service.cc:433] P 8cc13f87df2343dcb1c50aae26230e82: ending tablet copy session bb8d27ca78834bacb9ef4a67a829b494-b50c6f09335d4cb4a2c2288303fc1a7f on tablet b50c6f09335d4cb4a2c2288303fc1a7f with peer bb8d27ca78834bacb9ef4a67a829b494
I20240503 14:12:41.427680 24580 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:41.428062 24580 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: Bootstrap complete.
I20240503 14:12:41.428349 24580 ts_tablet_manager.cc:1367] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.002s
I20240503 14:12:41.428982 24580 raft_consensus.cc:348] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:41.429188 24580 raft_consensus.cc:729] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Initialized, Role: LEARNER
I20240503 14:12:41.429369 24580 consensus_queue.cc:260] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:41.430074 24580 ts_tablet_manager.cc:1398] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:41.430915 21342 tablet_copy_service.cc:342] P bce0dba863d04bcc8448b0d9f3ec710b: Request end of tablet copy session eb49440048bd4d749779867fa4ae1489-39aaa8ba3e0c43078f94e265a961fe97 received from {username='slave'} at 127.20.99.5:59839
I20240503 14:12:41.431064 21342 tablet_copy_service.cc:433] P bce0dba863d04bcc8448b0d9f3ec710b: ending tablet copy session eb49440048bd4d749779867fa4ae1489-39aaa8ba3e0c43078f94e265a961fe97 on tablet 39aaa8ba3e0c43078f94e265a961fe97 with peer eb49440048bd4d749779867fa4ae1489
I20240503 14:12:41.442912 22965 raft_consensus.cc:1203] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 14:12:41.470293 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:12:41.489738 22570 raft_consensus.cc:1203] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:41.515170 22703 raft_consensus.cc:1203] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240503 14:12:41.967422 24743 raft_consensus.cc:1050] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82: attempting to promote NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 to VOTER
I20240503 14:12:41.968106 24743 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:41.970809 22965 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:41.970912 22703 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:41.971259 21451 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:41.971439 24745 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:41.971889 24669 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:41.972261 24743 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:41.985205 24669 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:41.985778 22703 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:41.988236 22965 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:41.990516 20910 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d reported cstate change: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:41.991804 21451 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:41.997680 21192 consensus_queue.cc:237] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:41.999717 22965 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:42.000267 22703 raft_consensus.cc:1261] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:42.000300 24669 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:42.000764 24743 consensus_queue.cc:1035] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:42.003047 24743 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:42.005776 22703 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:42.008150 22965 raft_consensus.cc:2932] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:42.013998 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b50c6f09335d4cb4a2c2288303fc1a7f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:42.014130 20910 catalog_manager.cc:5484] T b50c6f09335d4cb4a2c2288303fc1a7f P 748b2b25b1b94a1b94788e8ece4ecb6d reported cstate change: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:42.019136 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet b50c6f09335d4cb4a2c2288303fc1a7f with delete_type TABLET_DATA_TOMBSTONED (TS 20560106dcaf405bbb288e4e3e8affa6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35362
I20240503 14:12:42.019836 24759 tablet_replica.cc:330] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: stopping tablet replica
I20240503 14:12:42.019995 24759 raft_consensus.cc:2229] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:42.020320 24759 raft_consensus.cc:2258] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:42.021212 24759 ts_tablet_manager.cc:1875] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:42.026474 24759 ts_tablet_manager.cc:1888] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:12:42.026647 24759 log.cc:1196] T b50c6f09335d4cb4a2c2288303fc1a7f P 20560106dcaf405bbb288e4e3e8affa6: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/b50c6f09335d4cb4a2c2288303fc1a7f
I20240503 14:12:42.027354 20906 catalog_manager.cc:4830] TS 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461): tablet b50c6f09335d4cb4a2c2288303fc1a7f (table rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]) successfully deleted
I20240503 14:12:42.055354 24775 raft_consensus.cc:1050] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: attempting to promote NON_VOTER eb49440048bd4d749779867fa4ae1489 to VOTER
I20240503 14:12:42.056195 24775 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:42.059092 21192 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:42.059091 22570 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 1 LEARNER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:42.059706 21451 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:42.059772 24604 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:42.061004 24775 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:42.061507 24781 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:42.067392 24775 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:42.069348 22570 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:42.070950 21451 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:42.072618 21192 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:42.073298 20917 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 4 to 5, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:42.110780 24776 raft_consensus.cc:1050] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b: attempting to promote NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d to VOTER
I20240503 14:12:42.111853 24776 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:42.114336 22703 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:42.114565 22965 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:42.114604 21192 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:42.115161 24775 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:42.115543 24776 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:42.115808 24781 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:42.130628 24781 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.131572 22965 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.132295 21192 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.135561 22703 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.136139 20915 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 8 to 9, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:42.141757 21321 consensus_queue.cc:237] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:42.143635 22703 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:42.143766 22965 raft_consensus.cc:1261] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer bce0dba863d04bcc8448b0d9f3ec710b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:42.144119 24776 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:42.144443 24781 consensus_queue.cc:1035] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:42.146607 24776 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.147370 22703 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.147435 22965 raft_consensus.cc:2932] T 98f8323a5a1c45ec97a4ccbc937fb135 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:42.149742 20905 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98f8323a5a1c45ec97a4ccbc937fb135 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:42.151369 20910 catalog_manager.cc:5484] T 98f8323a5a1c45ec97a4ccbc937fb135 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 9 to 10, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:42.155093 21171 tablet_service.cc:1514] Processing DeleteTablet for tablet 98f8323a5a1c45ec97a4ccbc937fb135 with delete_type TABLET_DATA_TOMBSTONED (TS 8cc13f87df2343dcb1c50aae26230e82 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50222
I20240503 14:12:42.155877 24792 tablet_replica.cc:330] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: stopping tablet replica
I20240503 14:12:42.156136 24792 raft_consensus.cc:2229] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:42.156509 24792 raft_consensus.cc:2258] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:42.157282 24792 ts_tablet_manager.cc:1875] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:42.161666 24792 ts_tablet_manager.cc:1888] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:12:42.161765 24792 log.cc:1196] T 98f8323a5a1c45ec97a4ccbc937fb135 P 8cc13f87df2343dcb1c50aae26230e82: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/98f8323a5a1c45ec97a4ccbc937fb135
I20240503 14:12:42.162456 20904 catalog_manager.cc:4830] TS 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): tablet 98f8323a5a1c45ec97a4ccbc937fb135 (table rebalance_test_table_1 [id=428e08465ced42e5b48b8fa8979076f9]) successfully deleted
I20240503 14:12:42.292819 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97"
dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44642
I20240503 14:12:42.293090 21321 raft_consensus.cc:595] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:42.321810 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97"
dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44666
I20240503 14:12:42.322026 21321 raft_consensus.cc:595] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Received request to transfer leadership
W20240503 14:12:42.486210 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:12:42.539918 21321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97"
dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44684
I20240503 14:12:42.540153 21321 raft_consensus.cc:595] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:42.577162 24781 raft_consensus.cc:979] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: : Instructing follower 8cc13f87df2343dcb1c50aae26230e82 to start an election
I20240503 14:12:42.577348 24781 raft_consensus.cc:1067] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Signalling peer 8cc13f87df2343dcb1c50aae26230e82 to start an election
I20240503 14:12:42.578310 21192 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97"
dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
from {username='slave'} at 127.20.99.3:40651
I20240503 14:12:42.578550 21192 raft_consensus.cc:482] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:42.578673 21192 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:42.580814 21192 raft_consensus.cc:504] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:42.581815 21192 leader_election.cc:288] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 election: Requested vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159), eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925)
I20240503 14:12:42.582283 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:42.582513 21451 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:42.582476 22570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "eb49440048bd4d749779867fa4ae1489"
I20240503 14:12:42.582830 22570 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:42.582870 21321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39aaa8ba3e0c43078f94e265a961fe97" candidate_uuid: "8cc13f87df2343dcb1c50aae26230e82" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bce0dba863d04bcc8448b0d9f3ec710b"
I20240503 14:12:42.583076 21321 raft_consensus.cc:3032] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:42.583160 21321 raft_consensus.cc:729] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bce0dba863d04bcc8448b0d9f3ec710b, State: Running, Role: LEADER
I20240503 14:12:42.583565 21321 consensus_queue.cc:260] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:42.584040 21321 raft_consensus.cc:3037] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:42.584954 21451 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 2.
I20240503 14:12:42.585804 22570 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 2.
I20240503 14:12:42.586318 21125 leader_election.cc:302] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 8cc13f87df2343dcb1c50aae26230e82, eb49440048bd4d749779867fa4ae1489; no voters:
I20240503 14:12:42.586606 24669 raft_consensus.cc:2781] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:42.586762 21321 raft_consensus.cc:2454] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc13f87df2343dcb1c50aae26230e82 in term 2.
I20240503 14:12:42.586982 24669 raft_consensus.cc:686] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 LEADER]: Becoming Leader. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Running, Role: LEADER
I20240503 14:12:42.587323 24669 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:42.589789 20915 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: term changed from 1 to 2, leader changed from bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) to 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2). New cstate: current_term: 2 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:42.946593 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:12:42.984282 21321 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:42.984906 24669 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:43.039623 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:12:43.153847 22570 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:43.154444 24669 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:43.160557 21451 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:43.161170 24847 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:12:43.285465 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 14:12:43.286511 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98f8323a5a1c45ec97a4ccbc937fb135: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:43.616169 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:12:43.707490 21192 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:12:43.709435 21451 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:12:43.709486 22570 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:12:43.710057 24875 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:43.710381 24874 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:43.712632 24847 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:43.713363 22570 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:43.713373 21451 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:12:43.716591 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39aaa8ba3e0c43078f94e265a961fe97 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:43.718469 20917 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 5 to 7, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New cstate: current_term: 2 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:43.724712 21302 tablet_service.cc:1514] Processing DeleteTablet for tablet 39aaa8ba3e0c43078f94e265a961fe97 with delete_type TABLET_DATA_TOMBSTONED (TS bce0dba863d04bcc8448b0d9f3ec710b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43526
I20240503 14:12:43.725569 24881 tablet_replica.cc:330] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: stopping tablet replica
I20240503 14:12:43.726137 24881 raft_consensus.cc:2229] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:43.726437 24881 raft_consensus.cc:2258] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:43.727596 24881 ts_tablet_manager.cc:1875] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:43.732663 24881 ts_tablet_manager.cc:1888] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:43.732766 24881 log.cc:1196] T 39aaa8ba3e0c43078f94e265a961fe97 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/39aaa8ba3e0c43078f94e265a961fe97
I20240503 14:12:43.733453 20905 catalog_manager.cc:4830] TS bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159): tablet 39aaa8ba3e0c43078f94e265a961fe97 (table rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]) successfully deleted
W20240503 14:12:43.755004 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:12:43.825831 21451 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:43.828469 22570 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:43.828591 21321 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:43.829157 24894 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:12:43.829525 21385 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693): Couldn't send request to peer ad5e6850bbff47fd9ff2e14de5b48223. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e999dc5b3bb441ba372e97ac0d6dcb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:43.829598 24854 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:43.833843 21321 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:43.834789 22570 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:43.833128 24894 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
I20240503 14:12:43.840694 20917 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 7 to 8, NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } } }
W20240503 14:12:44.161279 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:12:44.355721 24914 ts_tablet_manager.cc:902] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:44.356946 24914 tablet_copy_client.cc:308] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:44.357661 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 from peer ad5e6850bbff47fd9ff2e14de5b48223 ({username='slave'} at 127.20.99.7:41475)
I20240503 14:12:44.357815 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 from peer ad5e6850bbff47fd9ff2e14de5b48223 at {username='slave'} at 127.20.99.7:41475: session id = ad5e6850bbff47fd9ff2e14de5b48223-0e999dc5b3bb441ba372e97ac0d6dcb0
I20240503 14:12:44.359745 21472 tablet_copy_source_session.cc:205] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:44.360853 24914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:44.364974 24914 tablet_copy_client.cc:783] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:44.365187 24914 tablet_copy_client.cc:647] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:44.366757 24918 tablet_copy_client.cc:1002] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:44.710525 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:12:44.889230 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:12:44.902055 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:12:45.499156 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:12:45.692595 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 14:12:45.693742 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:12:45.867275 24914 tablet_copy_client.cc:523] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:45.870113 24914 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap starting.
I20240503 14:12:45.903028 24914 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:45.903492 24914 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: Bootstrap complete.
I20240503 14:12:45.903757 24914 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: Time spent bootstrapping tablet: real 0.034s user 0.032s sys 0.004s
I20240503 14:12:45.904480 24914 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:45.904759 24914 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ad5e6850bbff47fd9ff2e14de5b48223, State: Initialized, Role: LEARNER
I20240503 14:12:45.905018 24914 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:12:45.905742 24914 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:45.906450 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-0e999dc5b3bb441ba372e97ac0d6dcb0 received from {username='slave'} at 127.20.99.7:41475
I20240503 14:12:45.906621 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session ad5e6850bbff47fd9ff2e14de5b48223-0e999dc5b3bb441ba372e97ac0d6dcb0 on tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with peer ad5e6850bbff47fd9ff2e14de5b48223
W20240503 14:12:46.042858 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:12:46.314707 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b50c6f09335d4cb4a2c2288303fc1a7f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:46.363521 22833 raft_consensus.cc:1203] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240503 14:12:46.388733 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:12:46.776963 24986 raft_consensus.cc:1050] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER ad5e6850bbff47fd9ff2e14de5b48223 to VOTER
I20240503 14:12:46.777957 24986 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:46.780551 21321 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:46.780615 22833 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:46.780831 22570 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:12:46.781227 24988 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:46.781672 24986 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:46.781981 24967 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:12:46.783325 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:12:46.785068 24986 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.785815 21321 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.786141 22570 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.789897 20915 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b reported cstate change: config changed from index 8 to 9, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.790927 22833 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } attrs { replace: true } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.815661 21451 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } }
I20240503 14:12:46.817555 22833 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:46.817656 22570 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:12:46.818246 24988 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:46.818569 24986 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:46.821177 24967 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.821800 22570 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.825740 22833 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.827261 20918 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 9 to 10, VOTER bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } }
I20240503 14:12:46.829372 20906 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:46.831218 21302 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with delete_type TABLET_DATA_TOMBSTONED (TS bce0dba863d04bcc8448b0d9f3ec710b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43526
I20240503 14:12:46.834373 25002 tablet_replica.cc:330] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: stopping tablet replica
I20240503 14:12:46.834578 25002 raft_consensus.cc:2229] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:46.834949 25002 raft_consensus.cc:2258] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:46.835889 25002 ts_tablet_manager.cc:1875] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:46.840579 25002 ts_tablet_manager.cc:1888] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:12:46.840703 25002 log.cc:1196] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P bce0dba863d04bcc8448b0d9f3ec710b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e999dc5b3bb441ba372e97ac0d6dcb0
I20240503 14:12:46.841567 20905 catalog_manager.cc:4830] TS bce0dba863d04bcc8448b0d9f3ec710b (127.20.99.3:43159): tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 (table rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]) successfully deleted
I20240503 14:12:47.131698 21431 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:47.162097 22683 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:47.182973 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:47.200959 21171 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:47.202839 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:47.225194 22551 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:47.233402 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:47.301152 21451 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:47.303800 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:47.303932 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:12:47.304338 24985 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:47.304666 24967 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:12:47.307543 24985 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:47.311476 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:47.313067 20915 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 6 to 7, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:47.315253 21386 consensus_peers.cc:485] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299): Couldn't send request to peer 748b2b25b1b94a1b94788e8ece4ecb6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8c3db38486c4d41a85551928c0f2202. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:47.316046 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:47.355666 21192 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:47.358887 22570 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:12:47.359237 22833 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:12:47.359730 21126 consensus_peers.cc:485] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 -> Peer bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487): Couldn't send request to peer bb8d27ca78834bacb9ef4a67a829b494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df4f572b770c4fdf8d91d4f449f579eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:47.360321 25061 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:12:47.360642 24874 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:12:47.363173 25062 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:47.363672 22570 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:47.370386 20918 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 10 to 11, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:47.381052 22833 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
W20240503 14:12:47.451957 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:12:47.686997 25087 ts_tablet_manager.cc:902] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:47.688366 25087 tablet_copy_client.cc:308] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:47.694440 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet b8c3db38486c4d41a85551928c0f2202 from peer 748b2b25b1b94a1b94788e8ece4ecb6d ({username='slave'} at 127.20.99.6:41949)
I20240503 14:12:47.694615 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet b8c3db38486c4d41a85551928c0f2202 from peer 748b2b25b1b94a1b94788e8ece4ecb6d at {username='slave'} at 127.20.99.6:41949: session id = 748b2b25b1b94a1b94788e8ece4ecb6d-b8c3db38486c4d41a85551928c0f2202
I20240503 14:12:47.696858 21472 tablet_copy_source_session.cc:205] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:47.698086 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c3db38486c4d41a85551928c0f2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:47.702420 25087 tablet_copy_client.cc:783] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:47.702667 25087 tablet_copy_client.cc:647] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:47.704334 25091 tablet_copy_client.cc:1002] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:47.835525 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:12:47.957732 25104 ts_tablet_manager.cc:902] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: Initiating tablet copy from peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:47.959319 25104 tablet_copy_client.cc:308] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:33731
I20240503 14:12:47.960019 21212 tablet_copy_service.cc:140] P 8cc13f87df2343dcb1c50aae26230e82: Received BeginTabletCopySession request for tablet df4f572b770c4fdf8d91d4f449f579eb from peer bb8d27ca78834bacb9ef4a67a829b494 ({username='slave'} at 127.20.99.8:47311)
I20240503 14:12:47.960158 21212 tablet_copy_service.cc:161] P 8cc13f87df2343dcb1c50aae26230e82: Beginning new tablet copy session on tablet df4f572b770c4fdf8d91d4f449f579eb from peer bb8d27ca78834bacb9ef4a67a829b494 at {username='slave'} at 127.20.99.8:47311: session id = bb8d27ca78834bacb9ef4a67a829b494-df4f572b770c4fdf8d91d4f449f579eb
I20240503 14:12:47.961943 21212 tablet_copy_source_session.cc:205] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:47.963052 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4f572b770c4fdf8d91d4f449f579eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:47.967159 25104 tablet_copy_client.cc:783] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:47.967417 25104 tablet_copy_client.cc:647] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:47.968874 25108 tablet_copy_client.cc:1002] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:48.070652 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:12:48.345242 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:12:48.516989 25134 consensus_queue.cc:579] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Leader has been unable to successfully communicate with peer 1114934d9a3544a9935d75504bc27ceb for more than 30 seconds (30.160s)
I20240503 14:12:48.803699 25103 consensus_queue.cc:579] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Leader has been unable to successfully communicate with peer 1114934d9a3544a9935d75504bc27ceb for more than 30 seconds (30.055s)
W20240503 14:12:48.937014 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:12:49.204768 25087 tablet_copy_client.cc:523] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:49.207657 25087 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap starting.
I20240503 14:12:49.237746 25087 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:49.238212 25087 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap complete.
I20240503 14:12:49.238507 25087 ts_tablet_manager.cc:1367] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.000s
I20240503 14:12:49.239241 25087 raft_consensus.cc:348] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:49.239506 25087 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 748b2b25b1b94a1b94788e8ece4ecb6d, State: Initialized, Role: LEARNER
I20240503 14:12:49.239718 25087 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:49.240396 25087 ts_tablet_manager.cc:1398] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:12:49.241153 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-b8c3db38486c4d41a85551928c0f2202 received from {username='slave'} at 127.20.99.6:41949
I20240503 14:12:49.241276 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-b8c3db38486c4d41a85551928c0f2202 on tablet b8c3db38486c4d41a85551928c0f2202 with peer 748b2b25b1b94a1b94788e8ece4ecb6d
W20240503 14:12:49.243568 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:12:49.260007 22702 raft_consensus.cc:1203] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:12:49.333550 25156 consensus_queue.cc:579] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Leader has been unable to successfully communicate with peer 1114934d9a3544a9935d75504bc27ceb for more than 30 seconds (30.388s)
I20240503 14:12:49.469225 25104 tablet_copy_client.cc:523] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:49.471825 25104 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap starting.
W20240503 14:12:49.473520 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 14:12:49.474673 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet df4f572b770c4fdf8d91d4f449f579eb: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:49.504942 25104 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:49.505352 25104 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap complete.
I20240503 14:12:49.505606 25104 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: Time spent bootstrapping tablet: real 0.034s user 0.029s sys 0.002s
I20240503 14:12:49.506243 25104 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:49.506466 25104 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb8d27ca78834bacb9ef4a67a829b494, State: Initialized, Role: LEARNER
I20240503 14:12:49.506668 25104 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:49.507416 25104 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:12:49.508164 21212 tablet_copy_service.cc:342] P 8cc13f87df2343dcb1c50aae26230e82: Request end of tablet copy session bb8d27ca78834bacb9ef4a67a829b494-df4f572b770c4fdf8d91d4f449f579eb received from {username='slave'} at 127.20.99.8:47311
I20240503 14:12:49.508276 21212 tablet_copy_service.cc:433] P 8cc13f87df2343dcb1c50aae26230e82: ending tablet copy session bb8d27ca78834bacb9ef4a67a829b494-df4f572b770c4fdf8d91d4f449f579eb on tablet df4f572b770c4fdf8d91d4f449f579eb with peer bb8d27ca78834bacb9ef4a67a829b494
W20240503 14:12:49.756676 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:12:49.757541 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:12:49.813999 25182 raft_consensus.cc:1050] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d to VOTER
I20240503 14:12:49.814659 25182 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:49.817276 22702 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:49.817373 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:49.817513 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:12:49.817814 25220 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:49.818190 25156 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:49.818441 25182 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:49.821511 25156 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:49.822325 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:49.826120 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:49.827004 20915 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 reported cstate change: config changed from index 7 to 8, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:49.839720 22702 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:49.961603 22965 raft_consensus.cc:1203] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:12:50.033330 21451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202"
dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35370
I20240503 14:12:50.033579 21451 raft_consensus.cc:595] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:12:50.109383 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:12:50.110332 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:12:50.250598 25220 raft_consensus.cc:979] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: : Instructing follower 748b2b25b1b94a1b94788e8ece4ecb6d to start an election
I20240503 14:12:50.250808 25220 raft_consensus.cc:1067] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Signalling peer 748b2b25b1b94a1b94788e8ece4ecb6d to start an election
I20240503 14:12:50.251791 22702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202"
dest_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d"
from {username='slave'} at 127.20.99.4:39137
I20240503 14:12:50.252076 22702 raft_consensus.cc:482] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:50.252240 22702 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.254736 22702 raft_consensus.cc:504] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:50.256831 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:50.256947 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:50.257089 21451 raft_consensus.cc:3032] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:50.257154 21192 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.257187 21451 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:50.257544 21451 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:50.257985 21451 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.260076 21192 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748b2b25b1b94a1b94788e8ece4ecb6d in term 2.
I20240503 14:12:50.260720 21451 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748b2b25b1b94a1b94788e8ece4ecb6d in term 2.
I20240503 14:12:50.260705 22702 leader_election.cc:288] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [CANDIDATE]: Term 2 election: Requested vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487)
I20240503 14:12:50.268728 22639 leader_election.cc:302] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, 748b2b25b1b94a1b94788e8ece4ecb6d, 8cc13f87df2343dcb1c50aae26230e82; no voters:
I20240503 14:12:50.269133 25228 raft_consensus.cc:2781] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:50.269587 25228 raft_consensus.cc:686] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 LEADER]: Becoming Leader. State: Replica: 748b2b25b1b94a1b94788e8ece4ecb6d, State: Running, Role: LEADER
I20240503 14:12:50.270138 25228 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:50.274264 20918 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d reported cstate change: term changed from 1 to 2, leader changed from 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) to 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6). New cstate: current_term: 2 leader_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:50.274613 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202" candidate_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bb8d27ca78834bacb9ef4a67a829b494"
I20240503 14:12:50.274927 22965 raft_consensus.cc:3037] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.275734 21451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8c3db38486c4d41a85551928c0f2202"
dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35394
I20240503 14:12:50.275959 21451 raft_consensus.cc:595] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 14:12:50.276059 21451 raft_consensus.cc:603] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 14:12:50.277812 22965 raft_consensus.cc:2454] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748b2b25b1b94a1b94788e8ece4ecb6d in term 2.
W20240503 14:12:50.378875 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:12:50.478269 25220 consensus_queue.cc:579] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Leader has been unable to successfully communicate with peer 1114934d9a3544a9935d75504bc27ceb for more than 30 seconds (30.217s)
I20240503 14:12:50.501230 25264 raft_consensus.cc:1050] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: attempting to promote NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 to VOTER
I20240503 14:12:50.502097 25264 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:50.504707 22965 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 1 LEARNER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:12:50.505138 22570 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:12:50.505322 22833 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:12:50.505833 25277 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:50.506179 25062 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:50.506733 25277 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:50.517724 25279 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:50.518641 22570 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:50.522342 22833 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:50.523723 20911 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 11 to 12, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:50.523950 22965 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:50.541532 25157 consensus_queue.cc:579] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b [LEADER]: Leader has been unable to successfully communicate with peer 1114934d9a3544a9935d75504bc27ceb for more than 30 seconds (30.218s)
I20240503 14:12:50.556618 25265 consensus_queue.cc:579] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Leader has been unable to successfully communicate with peer 1114934d9a3544a9935d75504bc27ceb for more than 30 seconds (30.098s)
W20240503 14:12:50.558468 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:12:50.657290 21451 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:12:50.658007 25228 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:50.720196 21192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb"
dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47766
I20240503 14:12:50.720474 21192 raft_consensus.cc:595] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:50.779389 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:12:50.780053 25228 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:50.784463 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:12:50.785467 25228 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:12:50.882252 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 14:12:50.915124 25277 raft_consensus.cc:979] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: : Instructing follower eb49440048bd4d749779867fa4ae1489 to start an election
I20240503 14:12:50.915334 25277 raft_consensus.cc:1067] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Signalling peer eb49440048bd4d749779867fa4ae1489 to start an election
I20240503 14:12:50.916211 22570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb"
dest_uuid: "eb49440048bd4d749779867fa4ae1489"
from {username='slave'} at 127.20.99.2:33767
I20240503 14:12:50.916494 22570 raft_consensus.cc:482] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:50.916604 22570 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.918512 22570 raft_consensus.cc:504] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:50.920058 22570 leader_election.cc:288] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [CANDIDATE]: Term 2 election: Requested vote from peers 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731), ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693), bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487)
I20240503 14:12:50.920475 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "eb49440048bd4d749779867fa4ae1489" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "8cc13f87df2343dcb1c50aae26230e82"
I20240503 14:12:50.920677 21192 raft_consensus.cc:3032] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:50.920781 21192 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8cc13f87df2343dcb1c50aae26230e82, State: Running, Role: LEADER
I20240503 14:12:50.921176 21192 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:50.921767 21192 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.925539 21192 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49440048bd4d749779867fa4ae1489 in term 2.
I20240503 14:12:50.933848 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "eb49440048bd4d749779867fa4ae1489" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "bb8d27ca78834bacb9ef4a67a829b494"
I20240503 14:12:50.933835 22833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4f572b770c4fdf8d91d4f449f579eb" candidate_uuid: "eb49440048bd4d749779867fa4ae1489" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "ad5e6850bbff47fd9ff2e14de5b48223"
I20240503 14:12:50.934060 22965 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.934185 22833 raft_consensus.cc:3037] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:50.937014 22965 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49440048bd4d749779867fa4ae1489 in term 2.
I20240503 14:12:50.937103 22833 raft_consensus.cc:2454] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49440048bd4d749779867fa4ae1489 in term 2.
I20240503 14:12:50.937490 22505 leader_election.cc:302] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8cc13f87df2343dcb1c50aae26230e82, bb8d27ca78834bacb9ef4a67a829b494, eb49440048bd4d749779867fa4ae1489; no voters:
I20240503 14:12:50.937927 25284 raft_consensus.cc:2781] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:50.938916 25284 raft_consensus.cc:686] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 2 LEADER]: Becoming Leader. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Running, Role: LEADER
I20240503 14:12:50.939560 25284 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:50.943645 20911 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 reported cstate change: term changed from 1 to 2, leader changed from 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) to eb49440048bd4d749779867fa4ae1489 (127.20.99.5). New cstate: current_term: 2 leader_uuid: "eb49440048bd4d749779867fa4ae1489" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:51.206871 22702 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:51.208802 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:12:51.208788 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:12:51.209362 25305 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:51.209715 25228 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:51.212163 25303 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:51.212828 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:51.212860 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:51.217231 20911 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 reported cstate change: config changed from index 8 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:51.217762 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8c3db38486c4d41a85551928c0f2202 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:51.220947 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c3db38486c4d41a85551928c0f2202 with delete_type TABLET_DATA_TOMBSTONED (TS 20560106dcaf405bbb288e4e3e8affa6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35362
I20240503 14:12:51.221578 25338 tablet_replica.cc:330] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: stopping tablet replica
I20240503 14:12:51.221743 25338 raft_consensus.cc:2229] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:51.221932 25338 raft_consensus.cc:2258] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:51.222707 25338 ts_tablet_manager.cc:1875] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:51.227072 25338 ts_tablet_manager.cc:1888] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:12:51.227200 25338 log.cc:1196] T b8c3db38486c4d41a85551928c0f2202 P 20560106dcaf405bbb288e4e3e8affa6: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/b8c3db38486c4d41a85551928c0f2202
I20240503 14:12:51.227932 20906 catalog_manager.cc:4830] TS 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461): tablet b8c3db38486c4d41a85551928c0f2202 (table rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]) successfully deleted
I20240503 14:12:51.393700 22965 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:12:51.394429 25284 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:12:51.405229 21451 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:51.407862 22833 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240503 14:12:51.408051 21386 consensus_peers.cc:485] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299): Couldn't send request to peer 748b2b25b1b94a1b94788e8ece4ecb6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e999dc5b3bb441ba372e97ac0d6dcb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:51.408459 22570 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:12:51.408485 25356 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:12:51.408941 25265 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:12:51.411084 25356 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:51.411651 22833 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:51.412261 22570 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
I20240503 14:12:51.416839 20915 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 10 to 11, NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } } }
W20240503 14:12:51.468907 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:12:51.527472 21192 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:12:51.528213 25284 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:12:51.536317 22833 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:12:51.536962 25374 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 14:12:51.858295 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:12:51.953758 25404 ts_tablet_manager.cc:902] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: Initiating tablet copy from peer 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461)
I20240503 14:12:51.955206 25404 tablet_copy_client.cc:308] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:42461
I20240503 14:12:51.956089 21472 tablet_copy_service.cc:140] P 20560106dcaf405bbb288e4e3e8affa6: Received BeginTabletCopySession request for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 from peer 748b2b25b1b94a1b94788e8ece4ecb6d ({username='slave'} at 127.20.99.6:41949)
I20240503 14:12:51.956240 21472 tablet_copy_service.cc:161] P 20560106dcaf405bbb288e4e3e8affa6: Beginning new tablet copy session on tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 from peer 748b2b25b1b94a1b94788e8ece4ecb6d at {username='slave'} at 127.20.99.6:41949: session id = 748b2b25b1b94a1b94788e8ece4ecb6d-0e999dc5b3bb441ba372e97ac0d6dcb0
I20240503 14:12:51.958225 21472 tablet_copy_source_session.cc:205] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:51.959448 25404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.963534 25404 tablet_copy_client.cc:783] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:51.963766 25404 tablet_copy_client.cc:647] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:51.965240 25408 tablet_copy_client.cc:1002] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Injecting latency into file download: 1500
I20240503 14:12:52.066852 22570 consensus_queue.cc:237] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:52.068796 22965 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 14:12:52.068780 22833 raft_consensus.cc:1261] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 14:12:52.069439 25409 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 14:12:52.069804 25410 consensus_queue.cc:1035] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 14:12:52.072217 25284 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:52.072822 22965 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:52.072984 22833 raft_consensus.cc:2932] T df4f572b770c4fdf8d91d4f449f579eb P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:52.076418 20903 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df4f572b770c4fdf8d91d4f449f579eb with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:52.078256 20915 catalog_manager.cc:5484] T df4f572b770c4fdf8d91d4f449f579eb P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 12 to 14, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "eb49440048bd4d749779867fa4ae1489" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:52.082682 21171 tablet_service.cc:1514] Processing DeleteTablet for tablet df4f572b770c4fdf8d91d4f449f579eb with delete_type TABLET_DATA_TOMBSTONED (TS 8cc13f87df2343dcb1c50aae26230e82 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:50222
I20240503 14:12:52.083369 25417 tablet_replica.cc:330] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: stopping tablet replica
I20240503 14:12:52.083557 25417 raft_consensus.cc:2229] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:52.083784 25417 raft_consensus.cc:2258] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:52.084616 25417 ts_tablet_manager.cc:1875] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:52.089597 25417 ts_tablet_manager.cc:1888] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 14:12:52.089722 25417 log.cc:1196] T df4f572b770c4fdf8d91d4f449f579eb P 8cc13f87df2343dcb1c50aae26230e82: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/df4f572b770c4fdf8d91d4f449f579eb
I20240503 14:12:52.090500 20904 catalog_manager.cc:4830] TS 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): tablet df4f572b770c4fdf8d91d4f449f579eb (table rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]) successfully deleted
W20240503 14:12:52.386865 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:12:52.716202 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:12:52.885861 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:12:53.209067 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:12:53.434978 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 943f587dd47e486f9be763a1c52bf43f: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
I20240503 14:12:53.465620 25404 tablet_copy_client.cc:523] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:53.468230 25404 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap starting.
I20240503 14:12:53.502404 25404 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:53.502753 25404 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: Bootstrap complete.
I20240503 14:12:53.502959 25404 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.003s
I20240503 14:12:53.503608 25404 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:53.503820 25404 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 748b2b25b1b94a1b94788e8ece4ecb6d, State: Initialized, Role: LEARNER
I20240503 14:12:53.504025 25404 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: NON_VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: true } }
I20240503 14:12:53.504537 25404 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:12:53.505396 21472 tablet_copy_service.cc:342] P 20560106dcaf405bbb288e4e3e8affa6: Request end of tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-0e999dc5b3bb441ba372e97ac0d6dcb0 received from {username='slave'} at 127.20.99.6:41949
I20240503 14:12:53.505570 21472 tablet_copy_service.cc:433] P 20560106dcaf405bbb288e4e3e8affa6: ending tablet copy session 748b2b25b1b94a1b94788e8ece4ecb6d-0e999dc5b3bb441ba372e97ac0d6dcb0 on tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with peer 748b2b25b1b94a1b94788e8ece4ecb6d
I20240503 14:12:53.592301 22702 raft_consensus.cc:1203] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240503 14:12:53.959230 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:12:53.997789 25489 raft_consensus.cc:1050] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: attempting to promote NON_VOTER 748b2b25b1b94a1b94788e8ece4ecb6d to VOTER
I20240503 14:12:53.998705 25489 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:54.001235 22702 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 LEARNER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:12:54.001504 22570 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:12:54.002172 22833 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Refusing update from remote peer 20560106dcaf405bbb288e4e3e8affa6: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240503 14:12:54.002077 25491 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:54.002650 25489 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:54.002944 25459 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:12:54.008461 25459 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.011921 22702 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.012127 22833 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.019953 20915 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 reported cstate change: config changed from index 11 to 12, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20560106dcaf405bbb288e4e3e8affa6" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.020838 22570 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.042520 21451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0"
dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35542
I20240503 14:12:54.042714 21451 raft_consensus.cc:595] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:54.259938 21451 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0"
dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35552
I20240503 14:12:54.260144 21451 raft_consensus.cc:595] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:54.432711 25491 raft_consensus.cc:979] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: : Instructing follower eb49440048bd4d749779867fa4ae1489 to start an election
I20240503 14:12:54.432937 25491 raft_consensus.cc:1067] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Signalling peer eb49440048bd4d749779867fa4ae1489 to start an election
I20240503 14:12:54.433571 22570 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0"
dest_uuid: "eb49440048bd4d749779867fa4ae1489"
from {username='slave'} at 127.20.99.4:59317
I20240503 14:12:54.433825 22570 raft_consensus.cc:482] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:54.433943 22570 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:54.435928 22570 raft_consensus.cc:504] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:54.436791 22570 leader_election.cc:288] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [CANDIDATE]: Term 2 election: Requested vote from peers 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461), ad5e6850bbff47fd9ff2e14de5b48223 (127.20.99.7:43693), 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299)
I20240503 14:12:54.437343 22833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "eb49440048bd4d749779867fa4ae1489" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "ad5e6850bbff47fd9ff2e14de5b48223"
I20240503 14:12:54.437583 22833 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:54.437762 21451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "eb49440048bd4d749779867fa4ae1489" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "20560106dcaf405bbb288e4e3e8affa6"
I20240503 14:12:54.437965 21451 raft_consensus.cc:3032] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:54.438092 21451 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20560106dcaf405bbb288e4e3e8affa6, State: Running, Role: LEADER
I20240503 14:12:54.438386 21451 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:54.438869 21451 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:54.440104 22833 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49440048bd4d749779867fa4ae1489 in term 2.
I20240503 14:12:54.441655 21451 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49440048bd4d749779867fa4ae1489 in term 2.
I20240503 14:12:54.442075 22507 leader_election.cc:302] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 20560106dcaf405bbb288e4e3e8affa6, ad5e6850bbff47fd9ff2e14de5b48223, eb49440048bd4d749779867fa4ae1489; no voters:
I20240503 14:12:54.442350 25410 raft_consensus.cc:2781] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:54.442800 25410 raft_consensus.cc:686] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 2 LEADER]: Becoming Leader. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Running, Role: LEADER
I20240503 14:12:54.443132 25410 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:54.444802 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e999dc5b3bb441ba372e97ac0d6dcb0" candidate_uuid: "eb49440048bd4d749779867fa4ae1489" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d"
I20240503 14:12:54.445050 22702 raft_consensus.cc:3037] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:54.446403 20917 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 reported cstate change: term changed from 1 to 2, leader changed from 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) to eb49440048bd4d749779867fa4ae1489 (127.20.99.5). New cstate: current_term: 2 leader_uuid: "eb49440048bd4d749779867fa4ae1489" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:54.446962 22702 raft_consensus.cc:2454] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49440048bd4d749779867fa4ae1489 in term 2.
W20240503 14:12:54.521299 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:12:54.901197 22702 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:12:54.901835 25410 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 14:12:54.926060 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:12:54.964046 22833 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:12:54.964716 25410 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:12:54.969002 21451 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:12:54.970378 25410 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240503 14:12:54.973228 25544 consensus_queue.cc:1159] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "20560106dcaf405bbb288e4e3e8affa6" responder_term: 2 status { last_received { term: 2 index: 13 } last_committed_idx: 13 last_received_current_leader { term: 2 index: 13 } }
I20240503 14:12:54.973716 22570 consensus_queue.cc:237] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } }
I20240503 14:12:54.975903 22702 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 14:12:54.975924 22833 raft_consensus.cc:1261] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Refusing update from remote peer eb49440048bd4d749779867fa4ae1489: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240503 14:12:54.976408 25544 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 14:12:54.976701 25542 consensus_queue.cc:1035] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 14:12:54.978727 25542 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.979413 22833 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P ad5e6850bbff47fd9ff2e14de5b48223 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.980527 22702 raft_consensus.cc:2932] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } }
I20240503 14:12:54.982627 20903 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:54.984436 20911 catalog_manager.cc:5484] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 12 to 14, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "eb49440048bd4d749779867fa4ae1489" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:54.989074 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 with delete_type TABLET_DATA_TOMBSTONED (TS 20560106dcaf405bbb288e4e3e8affa6 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35362
I20240503 14:12:54.989692 25551 tablet_replica.cc:330] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: stopping tablet replica
I20240503 14:12:54.989923 25551 raft_consensus.cc:2229] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:54.990128 25551 raft_consensus.cc:2258] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:54.990828 25551 ts_tablet_manager.cc:1875] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:54.994982 25551 ts_tablet_manager.cc:1888] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240503 14:12:54.995079 25551 log.cc:1196] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 20560106dcaf405bbb288e4e3e8affa6: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e999dc5b3bb441ba372e97ac0d6dcb0
I20240503 14:12:54.995741 20906 catalog_manager.cc:4830] TS 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461): tablet 0e999dc5b3bb441ba372e97ac0d6dcb0 (table rebalance_test_table_0 [id=9c2b11a8ec4b4010b97628fbfefa2cba]) successfully deleted
W20240503 14:12:55.022087 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:12:55.112427 21192 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
W20240503 14:12:55.115041 21126 consensus_peers.cc:485] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8:40487): Couldn't send request to peer bb8d27ca78834bacb9ef4a67a829b494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39aaa8ba3e0c43078f94e265a961fe97. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:55.115099 22570 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:12:55.115387 21451 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:12:55.115664 25393 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:55.116008 25564 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:12:55.118921 25393 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:55.119458 22570 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:55.125088 20918 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New cstate: current_term: 2 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
I20240503 14:12:55.140022 21451 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } } }
W20240503 14:12:55.252473 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:12:55.319837 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 14:12:55.320816 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:12:55.433419 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:12:55.538030 25582 ts_tablet_manager.cc:902] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: Initiating tablet copy from peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731)
I20240503 14:12:55.539374 25582 tablet_copy_client.cc:308] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:33731
I20240503 14:12:55.540306 21212 tablet_copy_service.cc:140] P 8cc13f87df2343dcb1c50aae26230e82: Received BeginTabletCopySession request for tablet 39aaa8ba3e0c43078f94e265a961fe97 from peer bb8d27ca78834bacb9ef4a67a829b494 ({username='slave'} at 127.20.99.8:47311)
I20240503 14:12:55.540498 21212 tablet_copy_service.cc:161] P 8cc13f87df2343dcb1c50aae26230e82: Beginning new tablet copy session on tablet 39aaa8ba3e0c43078f94e265a961fe97 from peer bb8d27ca78834bacb9ef4a67a829b494 at {username='slave'} at 127.20.99.8:47311: session id = bb8d27ca78834bacb9ef4a67a829b494-39aaa8ba3e0c43078f94e265a961fe97
I20240503 14:12:55.542534 21212 tablet_copy_source_session.cc:205] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:55.543789 25582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39aaa8ba3e0c43078f94e265a961fe97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.548005 25582 tablet_copy_client.cc:783] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:55.548220 25582 tablet_copy_client.cc:647] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:55.549811 25586 tablet_copy_client.cc:1002] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:56.253536 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 14:12:56.400820 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:12:56.551429 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 14:12:56.552546 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0e999dc5b3bb441ba372e97ac0d6dcb0: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:56.941915 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:12:57.050185 25582 tablet_copy_client.cc:523] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:57.052978 25582 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap starting.
I20240503 14:12:57.084988 25582 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:57.085410 25582 tablet_bootstrap.cc:492] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: Bootstrap complete.
I20240503 14:12:57.085642 25582 ts_tablet_manager.cc:1367] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20240503 14:12:57.086228 25582 raft_consensus.cc:348] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:57.086445 25582 raft_consensus.cc:729] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: bb8d27ca78834bacb9ef4a67a829b494, State: Initialized, Role: LEARNER
I20240503 14:12:57.086601 25582 consensus_queue.cc:260] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: NON_VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: true } }
I20240503 14:12:57.087320 25582 ts_tablet_manager.cc:1398] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:12:57.088164 21212 tablet_copy_service.cc:342] P 8cc13f87df2343dcb1c50aae26230e82: Request end of tablet copy session bb8d27ca78834bacb9ef4a67a829b494-39aaa8ba3e0c43078f94e265a961fe97 received from {username='slave'} at 127.20.99.8:47311
I20240503 14:12:57.088279 21212 tablet_copy_service.cc:433] P 8cc13f87df2343dcb1c50aae26230e82: ending tablet copy session bb8d27ca78834bacb9ef4a67a829b494-39aaa8ba3e0c43078f94e265a961fe97 on tablet 39aaa8ba3e0c43078f94e265a961fe97 with peer bb8d27ca78834bacb9ef4a67a829b494
I20240503 14:12:57.371060 22965 raft_consensus.cc:1203] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240503 14:12:57.455936 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:12:57.509397 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:12:57.819622 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:12:57.906791 25656 raft_consensus.cc:1050] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82: attempting to promote NON_VOTER bb8d27ca78834bacb9ef4a67a829b494 to VOTER
W20240503 14:12:57.907254 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:12:57.907737 25656 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:57.910622 22570 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:12:57.910718 22965 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 LEARNER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:12:57.911319 25659 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:57.911523 21451 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:12:57.911712 25656 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:57.912081 25657 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:12:57.915334 25657 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.916034 22965 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.921399 20918 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 reported cstate change: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.921411 21451 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.922020 22570 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, bb8d27ca78834bacb9ef4a67a829b494 (127.20.99.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } attrs { replace: true } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.930776 21192 consensus_queue.cc:237] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } }
I20240503 14:12:57.932556 22570 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:12:57.932690 22965 raft_consensus.cc:1261] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer 8cc13f87df2343dcb1c50aae26230e82: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:12:57.933218 25605 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:57.933514 25659 consensus_queue.cc:1035] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:12:57.935658 25656 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.936439 22570 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.938548 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39aaa8ba3e0c43078f94e265a961fe97 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:57.939138 22965 raft_consensus.cc:2932] T 39aaa8ba3e0c43078f94e265a961fe97 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.940274 20915 catalog_manager.cc:5484] T 39aaa8ba3e0c43078f94e265a961fe97 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 9 to 10, VOTER 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "8cc13f87df2343dcb1c50aae26230e82" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } }
I20240503 14:12:57.944413 21431 tablet_service.cc:1514] Processing DeleteTablet for tablet 39aaa8ba3e0c43078f94e265a961fe97 with delete_type TABLET_DATA_TOMBSTONED (TS 20560106dcaf405bbb288e4e3e8affa6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35362
I20240503 14:12:57.945173 25668 tablet_replica.cc:330] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: stopping tablet replica
I20240503 14:12:57.945369 25668 raft_consensus.cc:2229] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:57.945580 25668 raft_consensus.cc:2258] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:57.946398 25668 ts_tablet_manager.cc:1875] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:57.950917 25668 ts_tablet_manager.cc:1888] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:12:57.951046 25668 log.cc:1196] T 39aaa8ba3e0c43078f94e265a961fe97 P 20560106dcaf405bbb288e4e3e8affa6: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/39aaa8ba3e0c43078f94e265a961fe97
I20240503 14:12:57.951818 20906 catalog_manager.cc:4830] TS 20560106dcaf405bbb288e4e3e8affa6 (127.20.99.4:42461): tablet 39aaa8ba3e0c43078f94e265a961fe97 (table rebalance_test_table_4 [id=2205f0b781d6459d89b071794fda5be9]) successfully deleted
I20240503 14:12:58.190549 22683 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:58.197799 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:58.206655 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:58.250560 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:58.265295 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:58.291584 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:58.297261 22551 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:58.374466 22702 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:12:58.377897 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:12:58.378613 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:12:58.378551 25530 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:12:58.379295 25530 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:12:58.381525 25718 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:58.382100 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:58.383405 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
W20240503 14:12:58.388443 22636 consensus_peers.cc:485] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d -> Peer eb49440048bd4d749779867fa4ae1489 (127.20.99.5:40925): Couldn't send request to peer eb49440048bd4d749779867fa4ae1489. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8c3db38486c4d41a85551928c0f2202. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:58.388698 20910 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 reported cstate change: config changed from index 10 to 11, NON_VOTER eb49440048bd4d749779867fa4ae1489 (127.20.99.5) added. New cstate: current_term: 2 leader_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } } }
I20240503 14:12:58.765486 25734 ts_tablet_manager.cc:902] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: Initiating tablet copy from peer 748b2b25b1b94a1b94788e8ece4ecb6d (127.20.99.6:36299)
I20240503 14:12:58.766774 25734 tablet_copy_client.cc:308] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:36299
I20240503 14:12:58.767676 22723 tablet_copy_service.cc:140] P 748b2b25b1b94a1b94788e8ece4ecb6d: Received BeginTabletCopySession request for tablet b8c3db38486c4d41a85551928c0f2202 from peer eb49440048bd4d749779867fa4ae1489 ({username='slave'} at 127.20.99.5:53895)
I20240503 14:12:58.767906 22723 tablet_copy_service.cc:161] P 748b2b25b1b94a1b94788e8ece4ecb6d: Beginning new tablet copy session on tablet b8c3db38486c4d41a85551928c0f2202 from peer eb49440048bd4d749779867fa4ae1489 at {username='slave'} at 127.20.99.5:53895: session id = eb49440048bd4d749779867fa4ae1489-b8c3db38486c4d41a85551928c0f2202
I20240503 14:12:58.770509 22723 tablet_copy_source_session.cc:205] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:58.772233 25734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c3db38486c4d41a85551928c0f2202. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.776796 25734 tablet_copy_client.cc:783] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:58.777099 25734 tablet_copy_client.cc:647] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:58.778950 25738 tablet_copy_client.cc:1002] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: tablet copy: Injecting latency into file download: 1500
W20240503 14:12:59.023247 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:12:59.585925 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:12:59.703238 20904 catalog_manager.cc:4573] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98f8323a5a1c45ec97a4ccbc937fb135: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111)
W20240503 14:12:59.834812 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:12:59.912062 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:13:00.017014 21386 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:13:00.279322 25734 tablet_copy_client.cc:523] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:00.281975 25734 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: Bootstrap starting.
I20240503 14:13:00.316668 25734 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:00.317176 25734 tablet_bootstrap.cc:492] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: Bootstrap complete.
I20240503 14:13:00.317440 25734 ts_tablet_manager.cc:1367] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.009s
I20240503 14:13:00.318101 25734 raft_consensus.cc:348] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:13:00.318329 25734 raft_consensus.cc:729] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eb49440048bd4d749779867fa4ae1489, State: Initialized, Role: LEARNER
I20240503 14:13:00.318533 25734 consensus_queue.cc:260] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: NON_VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: true } }
I20240503 14:13:00.319133 25734 ts_tablet_manager.cc:1398] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:00.320057 22723 tablet_copy_service.cc:342] P 748b2b25b1b94a1b94788e8ece4ecb6d: Request end of tablet copy session eb49440048bd4d749779867fa4ae1489-b8c3db38486c4d41a85551928c0f2202 received from {username='slave'} at 127.20.99.5:53895
I20240503 14:13:00.320241 22723 tablet_copy_service.cc:433] P 748b2b25b1b94a1b94788e8ece4ecb6d: ending tablet copy session eb49440048bd4d749779867fa4ae1489-b8c3db38486c4d41a85551928c0f2202 on tablet b8c3db38486c4d41a85551928c0f2202 with peer eb49440048bd4d749779867fa4ae1489
W20240503 14:13:00.346156 21256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 14:13:00.347002 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:13:00.650262 22570 raft_consensus.cc:1203] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240503 14:13:00.716403 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:13:01.253695 25813 raft_consensus.cc:1050] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d: attempting to promote NON_VOTER eb49440048bd4d749779867fa4ae1489 to VOTER
I20240503 14:13:01.254505 25813 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:13:01.257184 21192 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:13:01.257216 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:13:01.257544 22570 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 LEARNER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:13:01.257836 25816 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:13:01.258558 25813 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:13:01.258841 25753 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:13:01.264421 25816 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.265098 22570 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.265657 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.269786 21192 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.271328 20911 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 reported cstate change: config changed from index 11 to 12, eb49440048bd4d749779867fa4ae1489 (127.20.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } attrs { replace: true } } peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.296622 22702 consensus_queue.cc:237] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } }
I20240503 14:13:01.298391 22965 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:13:01.298519 22570 raft_consensus.cc:1261] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Refusing update from remote peer 748b2b25b1b94a1b94788e8ece4ecb6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 14:13:01.298892 25753 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:13:01.299255 25816 consensus_queue.cc:1035] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:13:01.301479 25753 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.302129 22965 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P bb8d27ca78834bacb9ef4a67a829b494 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.302721 22570 raft_consensus.cc:2932] T b8c3db38486c4d41a85551928c0f2202 P eb49440048bd4d749779867fa4ae1489 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } } }
I20240503 14:13:01.306450 20904 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8c3db38486c4d41a85551928c0f2202 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:01.307072 20910 catalog_manager.cc:5484] T b8c3db38486c4d41a85551928c0f2202 P 748b2b25b1b94a1b94788e8ece4ecb6d reported cstate change: config changed from index 12 to 13, VOTER 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "bb8d27ca78834bacb9ef4a67a829b494" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 40487 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "748b2b25b1b94a1b94788e8ece4ecb6d" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 36299 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb49440048bd4d749779867fa4ae1489" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 40925 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:01.310806 21171 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c3db38486c4d41a85551928c0f2202 with delete_type TABLET_DATA_TOMBSTONED (TS 8cc13f87df2343dcb1c50aae26230e82 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:50222
I20240503 14:13:01.311617 25827 tablet_replica.cc:330] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: stopping tablet replica
I20240503 14:13:01.311875 25827 raft_consensus.cc:2229] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:01.312273 25827 raft_consensus.cc:2258] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:01.313179 25827 ts_tablet_manager.cc:1875] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:01.318639 25827 ts_tablet_manager.cc:1888] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240503 14:13:01.318761 25827 log.cc:1196] T b8c3db38486c4d41a85551928c0f2202 P 8cc13f87df2343dcb1c50aae26230e82: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/b8c3db38486c4d41a85551928c0f2202
I20240503 14:13:01.322453 20904 catalog_manager.cc:4830] TS 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): tablet b8c3db38486c4d41a85551928c0f2202 (table rebalance_test_table_3 [id=bac04533c2c34a6d92fd0c60f362f4e0]) successfully deleted
I20240503 14:13:01.453922 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:01.459859 22683 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:01.475114 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:01.478613 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:01.493472 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:01.495684 22551 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:01.504207 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:01.692396 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:01.697407 22683 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:01.705336 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:01.709992 22551 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:01.714452 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:01.733723 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:01.743906 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:13:01.751309 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:13:01.811849 21126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 14:13:02.021862 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.1:43389
--local_ip_for_outbound_sockets=127.20.99.1
--tserver_master_addrs=127.20.99.62:39349
--webserver_port=34719
--webserver_interface=127.20.99.1
--builtin_ntp_servers=127.20.99.20:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 14:13:02.215880 21126 consensus_peers.cc:485] T 8ff7de43a9a048f7941ad0143d0b09f2 P 8cc13f87df2343dcb1c50aae26230e82 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:13:02.236028 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:13:02.364851 25905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:02.365166 25905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:02.365269 25905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:02.372957 25905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:02.373075 25905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:02.373242 25905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.1
I20240503 14:13:02.382503 25905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.1:43389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.99.1
--webserver_port=34719
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.99.62:39349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.1
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:02.384009 25905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:02.385864 25905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:02.399039 25911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:02.399891 25914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:02.400588 25912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:02.401346 25905 server_base.cc:1034] running on GCE node
I20240503 14:13:02.401883 25905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:02.402801 25905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:02.404006 25905 hybrid_clock.cc:648] HybridClock initialized: now 1714745582403927 us; error 53 us; skew 500 ppm
I20240503 14:13:02.404330 25905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:02.412613 25905 webserver.cc:460] Webserver started at http://127.20.99.1:34719/ using document root <none> and password file <none>
I20240503 14:13:02.413403 25905 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:02.413517 25905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:02.418823 25905 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:02.423022 25920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:02.424937 25905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
W20240503 14:13:02.424957 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:13:02.425156 25905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1114934d9a3544a9935d75504bc27ceb"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-p0gh"
I20240503 14:13:02.425969 25905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:02.466740 25905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:02.467671 25905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:02.467954 25905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:02.468895 25905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:02.470916 25927 ts_tablet_manager.cc:522] Loading tablet metadata (0/11 complete)
I20240503 14:13:02.493851 25905 ts_tablet_manager.cc:559] Loaded tablet metadata (11 total tablets, 11 live tablets)
I20240503 14:13:02.493978 25905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.023s user 0.001s sys 0.000s
I20240503 14:13:02.494148 25905 ts_tablet_manager.cc:574] Registering tablets (0/11 complete)
I20240503 14:13:02.497696 25927 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.514988 25905 ts_tablet_manager.cc:590] Registered 11 tablets
I20240503 14:13:02.515115 25905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.021s user 0.014s sys 0.003s
I20240503 14:13:02.517638 25927 log.cc:824] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:02.565400 25927 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.565979 25927 tablet_bootstrap.cc:492] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.567262 25927 ts_tablet_manager.cc:1367] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.070s user 0.061s sys 0.008s
I20240503 14:13:02.568261 25905 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.1:43389
I20240503 14:13:02.568311 26034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.1:43389 every 8 connection(s)
I20240503 14:13:02.570005 25905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:13:02.572206 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 25905
I20240503 14:13:02.575907 25927 raft_consensus.cc:348] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:13:02.577143 25927 raft_consensus.cc:729] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.577965 25927 consensus_queue.cc:260] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } }
I20240503 14:13:02.579766 25927 ts_tablet_manager.cc:1398] T 4cbc2976e019427c915aced7b0d4a50c P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240503 14:13:02.580199 25927 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.590171 26035 heartbeater.cc:347] Connected to a master server at 127.20.99.62:39349
I20240503 14:13:02.590559 26035 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:02.591796 26035 heartbeater.cc:511] Master 127.20.99.62:39349 requested a full tablet report, sending...
I20240503 14:13:02.603289 20910 ts_manager.cc:194] Re-registered known tserver with Master: 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389)
I20240503 14:13:02.611491 20910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.1:42843
I20240503 14:13:02.615882 26035 heartbeater.cc:503] Master 127.20.99.62:39349 was elected leader, sending a full tablet report...
I20240503 14:13:02.662456 25927 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.663976 25927 tablet_bootstrap.cc:492] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.665361 25927 ts_tablet_manager.cc:1367] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.085s user 0.046s sys 0.036s
I20240503 14:13:02.666481 25927 raft_consensus.cc:348] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:02.666913 25927 raft_consensus.cc:729] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.667316 25927 consensus_queue.cc:260] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:02.667949 25927 ts_tablet_manager.cc:1398] T 8ff7de43a9a048f7941ad0143d0b09f2 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:02.668303 25927 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.712836 25970 tablet_service.cc:1514] Processing DeleteTablet for tablet b50c6f09335d4cb4a2c2288303fc1a7f with delete_type TABLET_DATA_TOMBSTONED (TS 1114934d9a3544a9935d75504bc27ceb not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58988
W20240503 14:13:02.714509 20904 catalog_manager.cc:4810] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): delete failed for tablet b50c6f09335d4cb4a2c2288303fc1a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b50c6f09335d4cb4a2c2288303fc1a7f already in progress: opening tablet
I20240503 14:13:02.725382 25927 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.725965 25927 tablet_bootstrap.cc:492] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.727236 25927 ts_tablet_manager.cc:1367] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.059s user 0.042s sys 0.012s
I20240503 14:13:02.728061 25927 raft_consensus.cc:348] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:13:02.728297 25927 raft_consensus.cc:729] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.728559 25927 consensus_queue.cc:260] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:13:02.729072 25927 ts_tablet_manager.cc:1398] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:02.729367 25927 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.732057 25970 tablet_service.cc:1514] Processing DeleteTablet for tablet b50c6f09335d4cb4a2c2288303fc1a7f with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 10)) from {username='slave'} at 127.0.0.1:58988
I20240503 14:13:02.732476 26046 tablet_replica.cc:330] stopping tablet replica
I20240503 14:13:02.733012 26046 raft_consensus.cc:2229] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:02.733466 26046 raft_consensus.cc:2258] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:02.734722 26046 ts_tablet_manager.cc:1875] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:02.742100 26046 ts_tablet_manager.cc:1888] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:02.742254 26046 log.cc:1196] T b50c6f09335d4cb4a2c2288303fc1a7f P 1114934d9a3544a9935d75504bc27ceb: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/b50c6f09335d4cb4a2c2288303fc1a7f
I20240503 14:13:02.743369 20904 catalog_manager.cc:4830] TS 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): tablet b50c6f09335d4cb4a2c2288303fc1a7f (table rebalance_test_table_2 [id=43dd1d0f47d84c9284393e7dff764d7a]) successfully deleted
I20240503 14:13:02.775660 25927 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.776199 25927 tablet_bootstrap.cc:492] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.777226 25927 ts_tablet_manager.cc:1367] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.048s user 0.034s sys 0.011s
I20240503 14:13:02.777902 25927 raft_consensus.cc:348] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:13:02.778172 25927 raft_consensus.cc:729] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.778441 25927 consensus_queue.cc:260] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } attrs { replace: true } } peers { permanent_uuid: "ad5e6850bbff47fd9ff2e14de5b48223" member_type: NON_VOTER last_known_addr { host: "127.20.99.7" port: 43693 } attrs { promote: true } }
I20240503 14:13:02.778923 25927 ts_tablet_manager.cc:1398] T 943f587dd47e486f9be763a1c52bf43f P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:02.779206 25927 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.817987 25927 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.818497 25927 tablet_bootstrap.cc:492] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.819568 25927 ts_tablet_manager.cc:1367] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.015s
I20240503 14:13:02.820258 25927 raft_consensus.cc:348] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:13:02.820472 25927 raft_consensus.cc:729] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.820700 25927 consensus_queue.cc:260] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:13:02.821373 25927 ts_tablet_manager.cc:1398] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:02.821673 25927 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.861359 25927 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.861819 25927 tablet_bootstrap.cc:492] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.862820 25927 ts_tablet_manager.cc:1367] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.041s user 0.034s sys 0.004s
I20240503 14:13:02.863544 25927 raft_consensus.cc:348] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:13:02.863749 25927 raft_consensus.cc:729] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.864038 25927 consensus_queue.cc:260] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:13:02.864636 25927 ts_tablet_manager.cc:1398] T 0e999dc5b3bb441ba372e97ac0d6dcb0 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:13:02.865370 25927 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
W20240503 14:13:02.887252 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240503 14:13:02.902652 25927 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.903139 25927 tablet_bootstrap.cc:492] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.904136 25927 ts_tablet_manager.cc:1367] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.039s user 0.021s sys 0.016s
I20240503 14:13:02.904783 25927 raft_consensus.cc:348] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:02.905007 25927 raft_consensus.cc:729] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.905267 25927 consensus_queue.cc:260] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:02.905766 25927 ts_tablet_manager.cc:1398] T 1dca8761a296481cbdf63afd8a4d4fa1 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:02.906126 25927 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:02.948222 25927 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:02.948715 25927 tablet_bootstrap.cc:492] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:02.949700 25927 ts_tablet_manager.cc:1367] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.044s user 0.034s sys 0.007s
I20240503 14:13:02.950322 25927 raft_consensus.cc:348] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:02.950541 25927 raft_consensus.cc:729] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.950796 25927 consensus_queue.cc:260] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:02.951264 25927 ts_tablet_manager.cc:1398] T 98f8323a5a1c45ec97a4ccbc937fb135 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:13:02.951939 25927 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:03.001791 25927 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:03.002354 25927 tablet_bootstrap.cc:492] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:03.003700 25927 ts_tablet_manager.cc:1367] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.052s user 0.031s sys 0.020s
I20240503 14:13:03.004607 25927 raft_consensus.cc:348] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:13:03.005918 25927 raft_consensus.cc:729] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.006248 25927 consensus_queue.cc:260] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } } peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } }
I20240503 14:13:03.006877 25927 ts_tablet_manager.cc:1398] T df4f572b770c4fdf8d91d4f449f579eb P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:03.007139 25927 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
I20240503 14:13:03.047979 25927 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:03.048446 25927 tablet_bootstrap.cc:492] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:03.049429 25927 ts_tablet_manager.cc:1367] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.011s
I20240503 14:13:03.050143 25927 raft_consensus.cc:348] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:03.050307 25927 raft_consensus.cc:729] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.050550 25927 consensus_queue.cc:260] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "bce0dba863d04bcc8448b0d9f3ec710b" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43159 } }
I20240503 14:13:03.051110 25927 ts_tablet_manager.cc:1398] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:03.051481 25927 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap starting.
W20240503 14:13:03.096235 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20240503 14:13:03.100735 25927 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:03.101351 25927 tablet_bootstrap.cc:492] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Bootstrap complete.
I20240503 14:13:03.102533 25927 ts_tablet_manager.cc:1367] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Time spent bootstrapping tablet: real 0.051s user 0.035s sys 0.011s
I20240503 14:13:03.103435 25927 raft_consensus.cc:348] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:13:03.103744 25927 raft_consensus.cc:729] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1114934d9a3544a9935d75504bc27ceb, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.103996 25927 consensus_queue.cc:260] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cc13f87df2343dcb1c50aae26230e82" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 33731 } } peers { permanent_uuid: "1114934d9a3544a9935d75504bc27ceb" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 43389 } } peers { permanent_uuid: "20560106dcaf405bbb288e4e3e8affa6" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42461 } }
I20240503 14:13:03.104604 25927 ts_tablet_manager.cc:1398] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:03.168445 25970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.179911 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:03.183718 22683 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:03.214159 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.225512 25990 raft_consensus.cc:3037] T 8675b2b41d0d4058b286e73a43751a48 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:03.236733 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:03.283600 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.334700 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:03.359891 22551 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:03.455473 25990 raft_consensus.cc:3037] T de7258a94fee4aa49a19fe1436526abd P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:03.521981 25970 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.534734 22683 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:03.543493 21431 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:03.543923 22814 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.554427 21171 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:03.582412 22945 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:03.603704 21302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:03.648929 22551 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:03.681068 25990 raft_consensus.cc:3037] T bd6b3d66deae41439fcbf13cb5400045 P 1114934d9a3544a9935d75504bc27ceb [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:03.813944 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 25905
I20240503 14:13:03.821849 26029 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:03.933765 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 25905
W20240503 14:13:03.956104 21256 connection.cc:535] client connection to 127.20.99.1:43389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:03.956539 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21110
W20240503 14:13:03.958204 21256 consensus_peers.cc:485] T de7258a94fee4aa49a19fe1436526abd P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:03.962271 21232 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:13:03.969152 21256 consensus_peers.cc:485] T 1dca8761a296481cbdf63afd8a4d4fa1 P bce0dba863d04bcc8448b0d9f3ec710b -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:04.035392 21386 consensus_peers.cc:485] T 4cbc2976e019427c915aced7b0d4a50c P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:04.124397 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:04.155807 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21110
I20240503 14:13:04.188127 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21240
W20240503 14:13:04.188202 22898 connection.cc:535] server connection from 127.20.99.2:50333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:04.193760 21386 consensus_peers.cc:485] T 8675b2b41d0d4058b286e73a43751a48 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 1114934d9a3544a9935d75504bc27ceb (127.20.99.1:43389): Couldn't send request to peer 1114934d9a3544a9935d75504bc27ceb. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.1:43389: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:04.194902 21360 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:13:04.242720 21386 consensus_peers.cc:485] T a955820c3a79415c9d118901eb40f172 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): Couldn't send request to peer 8cc13f87df2343dcb1c50aae26230e82. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.2:33731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:04.289256 21386 consensus_peers.cc:485] T bd6b3d66deae41439fcbf13cb5400045 P 20560106dcaf405bbb288e4e3e8affa6 -> Peer 8cc13f87df2343dcb1c50aae26230e82 (127.20.99.2:33731): Couldn't send request to peer 8cc13f87df2343dcb1c50aae26230e82. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.2:33731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:04.376464 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21240
W20240503 14:13:04.408802 22638 connection.cc:535] server connection from 127.20.99.3:40277 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:04.409346 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 21370
W20240503 14:13:04.409499 20905 connection.cc:535] server connection from 127.20.99.3:47313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:04.415592 21490 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:04.606345 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 21370
W20240503 14:13:04.638854 20903 connection.cc:535] server connection from 127.20.99.4:42925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:04.639266 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 22489
I20240503 14:13:04.645964 22611 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:04.783696 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 22489
I20240503 14:13:04.809047 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 22620
I20240503 14:13:04.815251 22742 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:04.952970 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 22620
I20240503 14:13:04.978158 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 22751
I20240503 14:13:04.984485 22873 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:05.120270 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 22751
W20240503 14:13:05.144397 20903 connection.cc:535] server connection from 127.20.99.7:36761 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:05.144690 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 22882
I20240503 14:13:05.151430 23005 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:05.284263 26167 raft_consensus.cc:482] T b50c6f09335d4cb4a2c2288303fc1a7f P bb8d27ca78834bacb9ef4a67a829b494 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8cc13f87df2343dcb1c50aae26230e82)
I20240503 14:13:05.284500 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 22882
I20240503 14:13:05.308156 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 20888
I20240503 14:13:05.314427 20948 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:05.488404 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 20888
2024-05-03T14:13:05Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (69722 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (69722 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240503 14:13:05.561818 20876 test_util.cc:268] Using random seed: 1367470257
I20240503 14:13:05.563766 20876 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:13:05.563846 20876 ts_itest-base.cc:113] --------------
I20240503 14:13:05.563907 20876 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:13:05.563973 20876 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:13:05.564024 20876 ts_itest-base.cc:116] --------------
2024-05-03T14:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:13:05Z Disabled control of system clock
I20240503 14:13:05.589769 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:37747
--webserver_interface=127.20.99.62
--webserver_port=0
--builtin_ntp_servers=127.20.99.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.99.62:37747
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 14:13:05.919862 26177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:05.920208 26177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:05.920343 26177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:05.927959 26177 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:05.928125 26177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:13:05.928203 26177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:05.928296 26177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:13:05.928367 26177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:13:05.937685 26177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.99.62:37747
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.99.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:05.939059 26177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:05.940817 26177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:05.952172 26182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.952737 26185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:05.953439 26183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:05.954571 26177 server_base.cc:1034] running on GCE node
I20240503 14:13:05.955425 26177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:05.956933 26177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:05.958158 26177 hybrid_clock.cc:648] HybridClock initialized: now 1714745585958123 us; error 29 us; skew 500 ppm
I20240503 14:13:05.958503 26177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:05.960794 26177 webserver.cc:460] Webserver started at http://127.20.99.62:41947/ using document root <none> and password file <none>
I20240503 14:13:05.961511 26177 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:05.961627 26177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:05.961966 26177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:05.964658 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9289c2a91e4a451dbd1b9ad30b122534"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-p0gh"
I20240503 14:13:05.965435 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9289c2a91e4a451dbd1b9ad30b122534"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-p0gh"
I20240503 14:13:05.970402 26177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 14:13:05.973618 26191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:05.974797 26177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:05.974956 26177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9289c2a91e4a451dbd1b9ad30b122534"
format_stamp: "Formatted at 2024-05-03 14:13:05 on dist-test-slave-p0gh"
I20240503 14:13:05.975149 26177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:06.007292 26177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:06.008158 26177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:06.008425 26177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:06.026273 26177 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.62:37747
I20240503 14:13:06.026280 26242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.62:37747 every 8 connection(s)
I20240503 14:13:06.027709 26177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:13:06.031491 26243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.037744 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 26177
I20240503 14:13:06.037978 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:13:06.040983 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534: Bootstrap starting.
I20240503 14:13:06.044260 26243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.045583 26243 log.cc:824] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:06.048930 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534: No bootstrap required, opened a new log
I20240503 14:13:06.055781 26243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9289c2a91e4a451dbd1b9ad30b122534" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 37747 } }
I20240503 14:13:06.056126 26243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.056237 26243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9289c2a91e4a451dbd1b9ad30b122534, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.056792 26243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9289c2a91e4a451dbd1b9ad30b122534" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 37747 } }
I20240503 14:13:06.056994 26243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:13:06.057113 26243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:13:06.057262 26243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.059545 26243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9289c2a91e4a451dbd1b9ad30b122534" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 37747 } }
I20240503 14:13:06.060082 26243 leader_election.cc:302] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9289c2a91e4a451dbd1b9ad30b122534; no voters:
I20240503 14:13:06.060696 26243 leader_election.cc:288] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:13:06.060817 26248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:06.061583 26248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [term 1 LEADER]: Becoming Leader. State: Replica: 9289c2a91e4a451dbd1b9ad30b122534, State: Running, Role: LEADER
I20240503 14:13:06.062115 26248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9289c2a91e4a451dbd1b9ad30b122534" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 37747 } }
I20240503 14:13:06.062798 26243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:13:06.065227 26250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9289c2a91e4a451dbd1b9ad30b122534. Latest consensus state: current_term: 1 leader_uuid: "9289c2a91e4a451dbd1b9ad30b122534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9289c2a91e4a451dbd1b9ad30b122534" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 37747 } } }
I20240503 14:13:06.065407 26249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9289c2a91e4a451dbd1b9ad30b122534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9289c2a91e4a451dbd1b9ad30b122534" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 37747 } } }
I20240503 14:13:06.065629 26250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:06.065646 26249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:06.067898 26258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:13:06.074570 26258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:13:06.083577 26258 catalog_manager.cc:1347] Generated new cluster ID: 8974056c424c43009221de4aade95c02
I20240503 14:13:06.083695 26258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:13:06.106043 26258 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:13:06.107334 26258 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:13:06.120435 26258 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9289c2a91e4a451dbd1b9ad30b122534: Generated new TSK 0
I20240503 14:13:06.121302 26258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:13:06.127331 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.1:0
--local_ip_for_outbound_sockets=127.20.99.1
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--builtin_ntp_servers=127.20.99.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:13:06.458109 26267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:06.458420 26267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:06.458544 26267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:06.466745 26267 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:06.466862 26267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:06.467025 26267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.1
I20240503 14:13:06.476094 26267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.1
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:06.477409 26267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:06.479123 26267 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:06.492900 26272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:06.493084 26273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:06.493567 26275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:06.494660 26267 server_base.cc:1034] running on GCE node
I20240503 14:13:06.495357 26267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:06.496356 26267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:06.497596 26267 hybrid_clock.cc:648] HybridClock initialized: now 1714745586497552 us; error 56 us; skew 500 ppm
I20240503 14:13:06.497926 26267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:06.500288 26267 webserver.cc:460] Webserver started at http://127.20.99.1:34693/ using document root <none> and password file <none>
I20240503 14:13:06.501024 26267 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:06.501152 26267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:06.501470 26267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:06.504117 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bc1b5cf54934412aa5d3a210d80ca6a6"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-p0gh"
I20240503 14:13:06.504874 26267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bc1b5cf54934412aa5d3a210d80ca6a6"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-p0gh"
I20240503 14:13:06.510160 26267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:13:06.513759 26281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.515313 26267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:06.515498 26267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bc1b5cf54934412aa5d3a210d80ca6a6"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-p0gh"
I20240503 14:13:06.515704 26267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:06.538739 26267 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:06.539657 26267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:06.539921 26267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:06.540858 26267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:06.542296 26267 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:06.542399 26267 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.542510 26267 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:06.542582 26267 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.574792 26267 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.1:42965
I20240503 14:13:06.574862 26393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.1:42965 every 8 connection(s)
I20240503 14:13:06.576162 26267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:13:06.578132 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 26267
I20240503 14:13:06.578349 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:13:06.581845 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.2:0
--local_ip_for_outbound_sockets=127.20.99.2
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--builtin_ntp_servers=127.20.99.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:06.589612 26394 heartbeater.cc:347] Connected to a master server at 127.20.99.62:37747
I20240503 14:13:06.589922 26394 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:06.590677 26394 heartbeater.cc:511] Master 127.20.99.62:37747 requested a full tablet report, sending...
I20240503 14:13:06.592811 26208 ts_manager.cc:194] Registered new tserver with Master: bc1b5cf54934412aa5d3a210d80ca6a6 (127.20.99.1:42965)
I20240503 14:13:06.594274 26208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.1:57729
W20240503 14:13:06.903707 26398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:06.904031 26398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:06.904174 26398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:06.911747 26398 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:06.911903 26398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:06.912068 26398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.2
I20240503 14:13:06.921267 26398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.2
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:06.922657 26398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:06.924458 26398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:06.936918 26403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:06.937808 26406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:06.938469 26404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:06.939291 26398 server_base.cc:1034] running on GCE node
I20240503 14:13:06.939865 26398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:06.940790 26398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:06.942009 26398 hybrid_clock.cc:648] HybridClock initialized: now 1714745586941973 us; error 28 us; skew 500 ppm
I20240503 14:13:06.942356 26398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:06.944712 26398 webserver.cc:460] Webserver started at http://127.20.99.2:44193/ using document root <none> and password file <none>
I20240503 14:13:06.945466 26398 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:06.945581 26398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:06.945887 26398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:06.948599 26398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1d3f9c15f2af43368ab343bb7b1bcf74"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-p0gh"
I20240503 14:13:06.949472 26398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1d3f9c15f2af43368ab343bb7b1bcf74"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-p0gh"
I20240503 14:13:06.954700 26398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:13:06.958223 26412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.959652 26398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:06.959815 26398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1d3f9c15f2af43368ab343bb7b1bcf74"
format_stamp: "Formatted at 2024-05-03 14:13:06 on dist-test-slave-p0gh"
I20240503 14:13:06.960016 26398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:06.978005 26398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:06.979058 26398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:06.979389 26398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:06.980465 26398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:06.981918 26398 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:06.982015 26398 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:06.982137 26398 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:06.982209 26398 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.015234 26398 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.2:35625
I20240503 14:13:07.015308 26524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.2:35625 every 8 connection(s)
I20240503 14:13:07.016582 26398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:13:07.020632 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 26398
I20240503 14:13:07.020814 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:13:07.023907 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.3:0
--local_ip_for_outbound_sockets=127.20.99.3
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--builtin_ntp_servers=127.20.99.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:07.030100 26525 heartbeater.cc:347] Connected to a master server at 127.20.99.62:37747
I20240503 14:13:07.030421 26525 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:07.031280 26525 heartbeater.cc:511] Master 127.20.99.62:37747 requested a full tablet report, sending...
I20240503 14:13:07.032795 26208 ts_manager.cc:194] Registered new tserver with Master: 1d3f9c15f2af43368ab343bb7b1bcf74 (127.20.99.2:35625)
I20240503 14:13:07.033922 26208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.2:60863
W20240503 14:13:07.351552 26528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:07.351836 26528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:07.351948 26528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:07.359402 26528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:07.359525 26528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:07.359681 26528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.3
I20240503 14:13:07.369118 26528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.3
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:07.370441 26528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:07.372217 26528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:07.384568 26534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:07.385177 26533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:07.385805 26536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:07.386864 26528 server_base.cc:1034] running on GCE node
I20240503 14:13:07.387570 26528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:07.388505 26528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:07.389739 26528 hybrid_clock.cc:648] HybridClock initialized: now 1714745587389677 us; error 53 us; skew 500 ppm
I20240503 14:13:07.390095 26528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:07.392367 26528 webserver.cc:460] Webserver started at http://127.20.99.3:33375/ using document root <none> and password file <none>
I20240503 14:13:07.393085 26528 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:07.393239 26528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:07.393551 26528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:07.398582 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ed5103a2451d48658a78eb6d5c371adf"
format_stamp: "Formatted at 2024-05-03 14:13:07 on dist-test-slave-p0gh"
I20240503 14:13:07.399461 26528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ed5103a2451d48658a78eb6d5c371adf"
format_stamp: "Formatted at 2024-05-03 14:13:07 on dist-test-slave-p0gh"
I20240503 14:13:07.404692 26528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:07.408311 26542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.409684 26528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:07.409847 26528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ed5103a2451d48658a78eb6d5c371adf"
format_stamp: "Formatted at 2024-05-03 14:13:07 on dist-test-slave-p0gh"
I20240503 14:13:07.410037 26528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:07.441046 26528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:07.441946 26528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:07.442226 26528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:07.443223 26528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:07.444715 26528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:07.444809 26528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.444943 26528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:07.445015 26528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.476716 26528 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.3:34867
I20240503 14:13:07.476768 26654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.3:34867 every 8 connection(s)
I20240503 14:13:07.478018 26528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:13:07.485656 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 26528
I20240503 14:13:07.485805 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:13:07.489212 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.4:0
--local_ip_for_outbound_sockets=127.20.99.4
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--builtin_ntp_servers=127.20.99.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:07.490612 26655 heartbeater.cc:347] Connected to a master server at 127.20.99.62:37747
I20240503 14:13:07.490953 26655 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:07.491930 26655 heartbeater.cc:511] Master 127.20.99.62:37747 requested a full tablet report, sending...
I20240503 14:13:07.493456 26208 ts_manager.cc:194] Registered new tserver with Master: ed5103a2451d48658a78eb6d5c371adf (127.20.99.3:34867)
I20240503 14:13:07.494227 26208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.3:60387
I20240503 14:13:07.597138 26394 heartbeater.cc:503] Master 127.20.99.62:37747 was elected leader, sending a full tablet report...
W20240503 14:13:07.822353 26658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:07.822810 26658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:07.822993 26658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:07.832320 26658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:07.832588 26658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:07.832831 26658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.4
I20240503 14:13:07.842553 26658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.4
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:07.843923 26658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:07.845736 26658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:07.859009 26663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:07.859716 26664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:07.860702 26666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:07.861613 26658 server_base.cc:1034] running on GCE node
I20240503 14:13:07.862257 26658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:07.863380 26658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:07.864598 26658 hybrid_clock.cc:648] HybridClock initialized: now 1714745587864552 us; error 44 us; skew 500 ppm
I20240503 14:13:07.864982 26658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:07.867453 26658 webserver.cc:460] Webserver started at http://127.20.99.4:37689/ using document root <none> and password file <none>
I20240503 14:13:07.868266 26658 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:07.868394 26658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:07.868719 26658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:07.871351 26658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "05122586a9e34a2892f4a018abc6f883"
format_stamp: "Formatted at 2024-05-03 14:13:07 on dist-test-slave-p0gh"
I20240503 14:13:07.872140 26658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "05122586a9e34a2892f4a018abc6f883"
format_stamp: "Formatted at 2024-05-03 14:13:07 on dist-test-slave-p0gh"
I20240503 14:13:07.877385 26658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:13:07.880901 26672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.882423 26658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:07.882612 26658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "05122586a9e34a2892f4a018abc6f883"
format_stamp: "Formatted at 2024-05-03 14:13:07 on dist-test-slave-p0gh"
I20240503 14:13:07.882815 26658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:07.898576 26658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:07.899477 26658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:07.899749 26658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:07.900717 26658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:07.902165 26658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:07.902261 26658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.902377 26658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:07.902452 26658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:07.934597 26658 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.4:42863
I20240503 14:13:07.934652 26784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.4:42863 every 8 connection(s)
I20240503 14:13:07.935935 26658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:13:07.938598 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 26658
I20240503 14:13:07.938793 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:13:07.942301 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:0
--local_ip_for_outbound_sockets=127.20.99.5
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--builtin_ntp_servers=127.20.99.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:07.949559 26785 heartbeater.cc:347] Connected to a master server at 127.20.99.62:37747
I20240503 14:13:07.949887 26785 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:07.950817 26785 heartbeater.cc:511] Master 127.20.99.62:37747 requested a full tablet report, sending...
I20240503 14:13:07.952419 26208 ts_manager.cc:194] Registered new tserver with Master: 05122586a9e34a2892f4a018abc6f883 (127.20.99.4:42863)
I20240503 14:13:07.953622 26208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.4:59061
I20240503 14:13:08.036854 26525 heartbeater.cc:503] Master 127.20.99.62:37747 was elected leader, sending a full tablet report...
W20240503 14:13:08.281715 26788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:08.281996 26788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:08.282080 26788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:08.289492 26788 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:08.289623 26788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:08.289741 26788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:13:08.298846 26788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:37747
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:08.300185 26788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:08.301847 26788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:08.314181 26793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:08.314774 26796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:08.315843 26794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:08.316843 26788 server_base.cc:1034] running on GCE node
I20240503 14:13:08.317468 26788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:08.318410 26788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:08.319607 26788 hybrid_clock.cc:648] HybridClock initialized: now 1714745588319547 us; error 51 us; skew 500 ppm
I20240503 14:13:08.319923 26788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:08.322319 26788 webserver.cc:460] Webserver started at http://127.20.99.5:44839/ using document root <none> and password file <none>
I20240503 14:13:08.323050 26788 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:08.323168 26788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:08.323520 26788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:08.326130 26788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5d89d8172fef41ac8410b818683d2118"
format_stamp: "Formatted at 2024-05-03 14:13:08 on dist-test-slave-p0gh"
I20240503 14:13:08.326882 26788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5d89d8172fef41ac8410b818683d2118"
format_stamp: "Formatted at 2024-05-03 14:13:08 on dist-test-slave-p0gh"
I20240503 14:13:08.332011 26788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240503 14:13:08.335363 26802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:08.336666 26788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:08.336817 26788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5d89d8172fef41ac8410b818683d2118"
format_stamp: "Formatted at 2024-05-03 14:13:08 on dist-test-slave-p0gh"
I20240503 14:13:08.337003 26788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:08.361670 26788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:08.362619 26788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:08.362892 26788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:08.364007 26788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:08.365588 26788 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:08.365687 26788 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:08.365808 26788 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:08.365880 26788 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:08.397428 26788 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:39431
I20240503 14:13:08.397500 26914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:39431 every 8 connection(s)
I20240503 14:13:08.398762 26788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:08.402761 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 26788
I20240503 14:13:08.402944 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:13:08.411375 26915 heartbeater.cc:347] Connected to a master server at 127.20.99.62:37747
I20240503 14:13:08.411690 26915 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:08.412421 26915 heartbeater.cc:511] Master 127.20.99.62:37747 requested a full tablet report, sending...
I20240503 14:13:08.413733 26208 ts_manager.cc:194] Registered new tserver with Master: 5d89d8172fef41ac8410b818683d2118 (127.20.99.5:39431)
I20240503 14:13:08.414543 26208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:59237
I20240503 14:13:08.418478 20876 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:13:08.434456 26208 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46348:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:13:08.467679 26850 tablet_service.cc:1467] Processing CreateTablet for tablet 76e9eedd3e434a4c883561b8682887f4 (DEFAULT_TABLE table=TestTable [id=ad5a4b153f6347baaea78b079d0e20fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:08.467679 26720 tablet_service.cc:1467] Processing CreateTablet for tablet 76e9eedd3e434a4c883561b8682887f4 (DEFAULT_TABLE table=TestTable [id=ad5a4b153f6347baaea78b079d0e20fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:08.469237 26850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e9eedd3e434a4c883561b8682887f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:08.469249 26720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e9eedd3e434a4c883561b8682887f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:08.472620 26460 tablet_service.cc:1467] Processing CreateTablet for tablet 76e9eedd3e434a4c883561b8682887f4 (DEFAULT_TABLE table=TestTable [id=ad5a4b153f6347baaea78b079d0e20fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:08.473764 26460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e9eedd3e434a4c883561b8682887f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:08.485730 26934 tablet_bootstrap.cc:492] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: Bootstrap starting.
I20240503 14:13:08.488696 26933 tablet_bootstrap.cc:492] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: Bootstrap starting.
I20240503 14:13:08.490118 26934 tablet_bootstrap.cc:654] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:08.491458 26934 log.cc:824] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:08.491531 26935 tablet_bootstrap.cc:492] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: Bootstrap starting.
I20240503 14:13:08.493870 26933 tablet_bootstrap.cc:654] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:08.494673 26935 tablet_bootstrap.cc:654] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:08.495054 26933 log.cc:824] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:08.495685 26935 log.cc:824] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:08.495837 26934 tablet_bootstrap.cc:492] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: No bootstrap required, opened a new log
I20240503 14:13:08.496117 26934 ts_tablet_manager.cc:1367] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240503 14:13:08.496616 26655 heartbeater.cc:503] Master 127.20.99.62:37747 was elected leader, sending a full tablet report...
I20240503 14:13:08.497910 26933 tablet_bootstrap.cc:492] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: No bootstrap required, opened a new log
I20240503 14:13:08.498232 26933 ts_tablet_manager.cc:1367] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:13:08.498782 26935 tablet_bootstrap.cc:492] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: No bootstrap required, opened a new log
I20240503 14:13:08.498998 26935 ts_tablet_manager.cc:1367] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:13:08.505120 26935 raft_consensus.cc:348] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.505151 26934 raft_consensus.cc:348] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.505481 26935 raft_consensus.cc:374] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:08.505604 26935 raft_consensus.cc:729] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d89d8172fef41ac8410b818683d2118, State: Initialized, Role: FOLLOWER
I20240503 14:13:08.505616 26934 raft_consensus.cc:374] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:08.505712 26934 raft_consensus.cc:729] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3f9c15f2af43368ab343bb7b1bcf74, State: Initialized, Role: FOLLOWER
I20240503 14:13:08.506278 26935 consensus_queue.cc:260] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.506477 26934 consensus_queue.cc:260] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.507382 26915 heartbeater.cc:503] Master 127.20.99.62:37747 was elected leader, sending a full tablet report...
I20240503 14:13:08.506981 26933 raft_consensus.cc:348] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.507529 26933 raft_consensus.cc:374] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:08.507639 26933 raft_consensus.cc:729] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05122586a9e34a2892f4a018abc6f883, State: Initialized, Role: FOLLOWER
I20240503 14:13:08.508024 26935 ts_tablet_manager.cc:1398] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: Time spent starting tablet: real 0.009s user 0.006s sys 0.004s
I20240503 14:13:08.508407 26933 consensus_queue.cc:260] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.510141 26934 ts_tablet_manager.cc:1398] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20240503 14:13:08.510632 26785 heartbeater.cc:503] Master 127.20.99.62:37747 was elected leader, sending a full tablet report...
I20240503 14:13:08.511052 26933 ts_tablet_manager.cc:1398] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20240503 14:13:08.518946 26526 tablet.cc:2369] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:08.540616 26939 raft_consensus.cc:482] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:08.540879 26939 raft_consensus.cc:504] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.542697 26939 leader_election.cc:288] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05122586a9e34a2892f4a018abc6f883 (127.20.99.4:42863), 1d3f9c15f2af43368ab343bb7b1bcf74 (127.20.99.2:35625)
I20240503 14:13:08.549638 26740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e9eedd3e434a4c883561b8682887f4" candidate_uuid: "5d89d8172fef41ac8410b818683d2118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05122586a9e34a2892f4a018abc6f883" is_pre_election: true
I20240503 14:13:08.549926 26480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e9eedd3e434a4c883561b8682887f4" candidate_uuid: "5d89d8172fef41ac8410b818683d2118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" is_pre_election: true
I20240503 14:13:08.550129 26740 raft_consensus.cc:2454] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d89d8172fef41ac8410b818683d2118 in term 0.
I20240503 14:13:08.550371 26480 raft_consensus.cc:2454] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d89d8172fef41ac8410b818683d2118 in term 0.
I20240503 14:13:08.550827 26804 leader_election.cc:302] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05122586a9e34a2892f4a018abc6f883, 5d89d8172fef41ac8410b818683d2118; no voters:
I20240503 14:13:08.551268 26939 raft_consensus.cc:2781] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:08.551421 26939 raft_consensus.cc:482] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:08.551515 26939 raft_consensus.cc:3037] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.553469 26939 raft_consensus.cc:504] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.554021 26939 leader_election.cc:288] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [CANDIDATE]: Term 1 election: Requested vote from peers 05122586a9e34a2892f4a018abc6f883 (127.20.99.4:42863), 1d3f9c15f2af43368ab343bb7b1bcf74 (127.20.99.2:35625)
I20240503 14:13:08.554476 26740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e9eedd3e434a4c883561b8682887f4" candidate_uuid: "5d89d8172fef41ac8410b818683d2118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05122586a9e34a2892f4a018abc6f883"
I20240503 14:13:08.554543 26480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e9eedd3e434a4c883561b8682887f4" candidate_uuid: "5d89d8172fef41ac8410b818683d2118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74"
I20240503 14:13:08.554683 26740 raft_consensus.cc:3037] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.554731 26480 raft_consensus.cc:3037] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.557762 26480 raft_consensus.cc:2454] T 76e9eedd3e434a4c883561b8682887f4 P 1d3f9c15f2af43368ab343bb7b1bcf74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d89d8172fef41ac8410b818683d2118 in term 1.
I20240503 14:13:08.557761 26740 raft_consensus.cc:2454] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d89d8172fef41ac8410b818683d2118 in term 1.
I20240503 14:13:08.558146 26804 leader_election.cc:302] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3f9c15f2af43368ab343bb7b1bcf74, 5d89d8172fef41ac8410b818683d2118; no voters:
I20240503 14:13:08.558439 26939 raft_consensus.cc:2781] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:08.558854 26939 raft_consensus.cc:686] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [term 1 LEADER]: Becoming Leader. State: Replica: 5d89d8172fef41ac8410b818683d2118, State: Running, Role: LEADER
I20240503 14:13:08.559464 26939 consensus_queue.cc:237] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } }
I20240503 14:13:08.563771 26207 catalog_manager.cc:5484] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d89d8172fef41ac8410b818683d2118 (127.20.99.5). New cstate: current_term: 1 leader_uuid: "5d89d8172fef41ac8410b818683d2118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 } } peers { permanent_uuid: "5d89d8172fef41ac8410b818683d2118" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 39431 } } peers { permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 } } }
I20240503 14:13:08.583024 20876 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:13:08.585620 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bc1b5cf54934412aa5d3a210d80ca6a6 to finish bootstrapping
I20240503 14:13:08.594755 20876 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1d3f9c15f2af43368ab343bb7b1bcf74 to finish bootstrapping
I20240503 14:13:08.602458 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ed5103a2451d48658a78eb6d5c371adf to finish bootstrapping
I20240503 14:13:08.610728 20876 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 05122586a9e34a2892f4a018abc6f883 to finish bootstrapping
I20240503 14:13:08.617581 20876 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5d89d8172fef41ac8410b818683d2118 to finish bootstrapping
W20240503 14:13:08.650503 26916 tablet.cc:2369] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:13:08.688035 26786 tablet.cc:2369] T 76e9eedd3e434a4c883561b8682887f4 P 05122586a9e34a2892f4a018abc6f883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:08.966910 26939 consensus_queue.cc:1035] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05122586a9e34a2892f4a018abc6f883" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 42863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.975111 26943 consensus_queue.cc:1035] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3f9c15f2af43368ab343bb7b1bcf74" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.088280 26460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.089404 26720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.093825 26850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:09.097460 26329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.098897 26590 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.260133 26720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.265823 26329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.270625 26590 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.276072 26460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.277061 26850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:09.871992 26460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.876209 26720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.877614 26850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:09.877648 26329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:09.892797 26590 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.049785 26329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.054055 26850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:10.054419 26460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.062264 26590 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.063696 26720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.224560 26720 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.232991 26329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.242549 26590 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.249600 26850 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:10.250300 26460 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:10.397913 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 26267
I20240503 14:13:10.399299 26388 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:10.494575 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 26267
I20240503 14:13:10.512697 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 26398
I20240503 14:13:10.513685 26519 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:10.609725 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 26398
I20240503 14:13:10.629703 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 26528
I20240503 14:13:10.630723 26649 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:10.729539 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 26528
I20240503 14:13:10.747514 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 26658
I20240503 14:13:10.748484 26780 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:10.842787 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 26658
W20240503 14:13:10.846477 26804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.2:35625: connect: Connection refused (error 111)
W20240503 14:13:10.848230 26804 consensus_peers.cc:485] T 76e9eedd3e434a4c883561b8682887f4 P 5d89d8172fef41ac8410b818683d2118 -> Peer 1d3f9c15f2af43368ab343bb7b1bcf74 (127.20.99.2:35625): Couldn't send request to peer 1d3f9c15f2af43368ab343bb7b1bcf74. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.2:35625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:10.861577 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 26788
I20240503 14:13:10.862625 26909 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:10.957193 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 26788
I20240503 14:13:10.975522 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 26177
I20240503 14:13:10.976656 26238 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:11.090723 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 26177
2024-05-03T14:13:11Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5572 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5572 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240503 14:13:11.134451 20876 test_util.cc:268] Using random seed: 1373042906
I20240503 14:13:11.136373 20876 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:13:11.136442 20876 ts_itest-base.cc:113] --------------
I20240503 14:13:11.136493 20876 ts_itest-base.cc:114] 7 tablet servers
I20240503 14:13:11.136545 20876 ts_itest-base.cc:115] 2 replicas per TS
I20240503 14:13:11.136605 20876 ts_itest-base.cc:116] --------------
2024-05-03T14:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:13:11Z Disabled control of system clock
I20240503 14:13:11.160928 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:38501
--webserver_interface=127.20.99.62
--webserver_port=0
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.99.62:38501
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 14:13:11.484239 27139 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:13:11.484587 27139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:11.484691 27139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:11.484807 27139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:11.492426 27139 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:11.492547 27139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:13:11.492646 27139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:11.492709 27139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:13:11.492792 27139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:13:11.502050 27139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.99.62:38501
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:38501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.99.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:11.503468 27139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:11.505255 27139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:11.516525 27144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:11.517401 27147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:11.518122 27145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:11.518900 27139 server_base.cc:1034] running on GCE node
I20240503 14:13:11.519762 27139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:11.521235 27139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:11.522461 27139 hybrid_clock.cc:648] HybridClock initialized: now 1714745591522389 us; error 55 us; skew 500 ppm
I20240503 14:13:11.522784 27139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:11.530544 27139 webserver.cc:460] Webserver started at http://127.20.99.62:45083/ using document root <none> and password file <none>
I20240503 14:13:11.531353 27139 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:11.531495 27139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:11.531802 27139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:11.534526 27139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "16e0966be88a4fdfbedfd01d9ae02384"
format_stamp: "Formatted at 2024-05-03 14:13:11 on dist-test-slave-p0gh"
I20240503 14:13:11.535378 27139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "16e0966be88a4fdfbedfd01d9ae02384"
format_stamp: "Formatted at 2024-05-03 14:13:11 on dist-test-slave-p0gh"
I20240503 14:13:11.540529 27139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:13:11.543869 27153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:11.545152 27139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:11.545312 27139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
uuid: "16e0966be88a4fdfbedfd01d9ae02384"
format_stamp: "Formatted at 2024-05-03 14:13:11 on dist-test-slave-p0gh"
I20240503 14:13:11.545492 27139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:11.565936 27139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:11.566756 27139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:11.567035 27139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:11.584847 27139 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.62:38501
I20240503 14:13:11.584873 27204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.62:38501 every 8 connection(s)
I20240503 14:13:11.586480 27139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:13:11.588711 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27139
I20240503 14:13:11.588891 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:13:11.590299 27205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:11.602456 27205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384: Bootstrap starting.
I20240503 14:13:11.605301 27205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:11.606240 27205 log.cc:824] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:11.608876 27205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384: No bootstrap required, opened a new log
I20240503 14:13:11.614578 27205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0966be88a4fdfbedfd01d9ae02384" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 38501 } }
I20240503 14:13:11.614898 27205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:11.614992 27205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e0966be88a4fdfbedfd01d9ae02384, State: Initialized, Role: FOLLOWER
I20240503 14:13:11.615655 27205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0966be88a4fdfbedfd01d9ae02384" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 38501 } }
I20240503 14:13:11.615895 27205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:13:11.616019 27205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:13:11.616171 27205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:11.618125 27205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0966be88a4fdfbedfd01d9ae02384" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 38501 } }
I20240503 14:13:11.618569 27205 leader_election.cc:302] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e0966be88a4fdfbedfd01d9ae02384; no voters:
I20240503 14:13:11.619082 27205 leader_election.cc:288] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:13:11.619262 27210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:11.619843 27210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [term 1 LEADER]: Becoming Leader. State: Replica: 16e0966be88a4fdfbedfd01d9ae02384, State: Running, Role: LEADER
I20240503 14:13:11.620424 27210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0966be88a4fdfbedfd01d9ae02384" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 38501 } }
I20240503 14:13:11.620929 27205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:13:11.623476 27211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e0966be88a4fdfbedfd01d9ae02384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0966be88a4fdfbedfd01d9ae02384" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 38501 } } }
I20240503 14:13:11.623673 27212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e0966be88a4fdfbedfd01d9ae02384. Latest consensus state: current_term: 1 leader_uuid: "16e0966be88a4fdfbedfd01d9ae02384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e0966be88a4fdfbedfd01d9ae02384" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 38501 } } }
I20240503 14:13:11.623839 27211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:11.623906 27212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384 [sys.catalog]: This master's current role is: LEADER
I20240503 14:13:11.625752 27218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:13:11.630890 27218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:13:11.648941 27218 catalog_manager.cc:1347] Generated new cluster ID: f2babfa73c54490e96e54ee5a9f2b941
I20240503 14:13:11.649089 27218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:13:11.678432 27218 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:13:11.680475 27218 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:13:11.693746 27218 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 16e0966be88a4fdfbedfd01d9ae02384: Generated new TSK 0
I20240503 14:13:11.694676 27218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:13:11.708648 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.1:0
--local_ip_for_outbound_sockets=127.20.99.1
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:13:12.048092 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:12.048395 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:12.048528 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:12.056164 27229 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:12.056298 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:12.056452 27229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.1
I20240503 14:13:12.065486 27229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.1
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:12.066854 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:12.068665 27229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:12.081029 27234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:12.081960 27235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:12.082731 27237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:12.083555 27229 server_base.cc:1034] running on GCE node
I20240503 14:13:12.084177 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:12.085153 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:12.086374 27229 hybrid_clock.cc:648] HybridClock initialized: now 1714745592086330 us; error 45 us; skew 500 ppm
I20240503 14:13:12.086699 27229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:12.089037 27229 webserver.cc:460] Webserver started at http://127.20.99.1:44609/ using document root <none> and password file <none>
I20240503 14:13:12.089768 27229 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:12.089890 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:12.090205 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:12.092923 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "60da896b8369489994355cc0e9050514"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.093869 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "60da896b8369489994355cc0e9050514"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.099203 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 14:13:12.102671 27243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.104151 27229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:12.104310 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "60da896b8369489994355cc0e9050514"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.104511 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:12.122792 27229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:12.123701 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:12.123979 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:12.124943 27229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:12.126551 27229 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:12.126649 27229 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.126763 27229 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:12.126834 27229 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.158174 27229 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.1:45565
I20240503 14:13:12.158223 27355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.1:45565 every 8 connection(s)
I20240503 14:13:12.159473 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:13:12.160087 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27229
I20240503 14:13:12.160373 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:13:12.163996 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.2:0
--local_ip_for_outbound_sockets=127.20.99.2
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:12.174692 27356 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:12.175012 27356 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:12.175827 27356 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:12.178206 27170 ts_manager.cc:194] Registered new tserver with Master: 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:12.180596 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.1:57161
W20240503 14:13:12.486346 27360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:12.486640 27360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:12.486729 27360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:12.494098 27360 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:12.494212 27360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:12.494374 27360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.2
I20240503 14:13:12.503154 27360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.2
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:12.504762 27360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:12.506776 27360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:12.519347 27366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:12.519917 27365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:12.520865 27368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:12.521755 27360 server_base.cc:1034] running on GCE node
I20240503 14:13:12.522351 27360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:12.523296 27360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:12.524518 27360 hybrid_clock.cc:648] HybridClock initialized: now 1714745592524476 us; error 52 us; skew 500 ppm
I20240503 14:13:12.524829 27360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:12.530913 27360 webserver.cc:460] Webserver started at http://127.20.99.2:35729/ using document root <none> and password file <none>
I20240503 14:13:12.531699 27360 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:12.531814 27360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:12.532126 27360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:12.536648 27360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "aad738f0c10848789c2c811160990f8b"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.537439 27360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "aad738f0c10848789c2c811160990f8b"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.542248 27360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:13:12.545452 27374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.546698 27360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:12.546847 27360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aad738f0c10848789c2c811160990f8b"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.547031 27360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:12.565030 27360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:12.565794 27360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:12.566052 27360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:12.566933 27360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:12.568344 27360 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:12.568439 27360 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.568547 27360 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:12.568616 27360 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.601316 27360 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.2:42553
I20240503 14:13:12.601351 27486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.2:42553 every 8 connection(s)
I20240503 14:13:12.602721 27360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:13:12.605871 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27360
I20240503 14:13:12.606029 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:13:12.609097 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.3:0
--local_ip_for_outbound_sockets=127.20.99.3
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:12.616429 27487 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:12.616752 27487 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:12.617676 27487 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:12.619163 27170 ts_manager.cc:194] Registered new tserver with Master: aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:12.620333 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.2:46671
W20240503 14:13:12.930694 27490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:12.930969 27490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:12.931053 27490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:12.938393 27490 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:12.938506 27490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:12.938625 27490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.3
I20240503 14:13:12.947432 27490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.3
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:12.948725 27490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:12.950448 27490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:12.962766 27495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:12.963618 27496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:12.964331 27498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:12.965163 27490 server_base.cc:1034] running on GCE node
I20240503 14:13:12.965727 27490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:12.966691 27490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:12.967882 27490 hybrid_clock.cc:648] HybridClock initialized: now 1714745592967824 us; error 50 us; skew 500 ppm
I20240503 14:13:12.968199 27490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:12.970321 27490 webserver.cc:460] Webserver started at http://127.20.99.3:33123/ using document root <none> and password file <none>
I20240503 14:13:12.971005 27490 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:12.971150 27490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:12.971475 27490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:12.973906 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.974656 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.979513 27490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240503 14:13:12.982666 27504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:12.983964 27490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:12.984150 27490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
format_stamp: "Formatted at 2024-05-03 14:13:12 on dist-test-slave-p0gh"
I20240503 14:13:12.984333 27490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:13.000675 27490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:13.001492 27490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:13.001768 27490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:13.002669 27490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:13.004141 27490 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:13.004238 27490 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.004354 27490 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:13.004427 27490 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.039114 27490 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.3:43621
I20240503 14:13:13.039209 27616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.3:43621 every 8 connection(s)
I20240503 14:13:13.040419 27490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:13:13.049459 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27490
I20240503 14:13:13.049681 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:13:13.052532 27617 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:13.052811 27617 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:13.053179 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.4:0
--local_ip_for_outbound_sockets=127.20.99.4
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:13.053575 27617 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:13.054991 27169 ts_manager.cc:194] Registered new tserver with Master: 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:13.056242 27169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.3:55993
I20240503 14:13:13.183702 27356 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
W20240503 14:13:13.381567 27620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:13.381906 27620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:13.382063 27620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:13.389519 27620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:13.389667 27620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:13.389822 27620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.4
I20240503 14:13:13.398705 27620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.4
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:13.400553 27620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:13.402297 27620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:13.414561 27626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:13.415247 27625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:13.415935 27628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:13.416886 27620 server_base.cc:1034] running on GCE node
I20240503 14:13:13.417439 27620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:13.418304 27620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:13.419535 27620 hybrid_clock.cc:648] HybridClock initialized: now 1714745593419483 us; error 35 us; skew 500 ppm
I20240503 14:13:13.419858 27620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:13.422086 27620 webserver.cc:460] Webserver started at http://127.20.99.4:41603/ using document root <none> and password file <none>
I20240503 14:13:13.422804 27620 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:13.422919 27620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:13.423264 27620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:13.425731 27620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c57eaae6a75a43679120a10c6d3090c7"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:13.426477 27620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c57eaae6a75a43679120a10c6d3090c7"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:13.431566 27620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:13:13.434746 27634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.436147 27620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:13.436293 27620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c57eaae6a75a43679120a10c6d3090c7"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:13.436470 27620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:13.451771 27620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:13.452600 27620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:13.452894 27620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:13.453830 27620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:13.455269 27620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:13.455364 27620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.455480 27620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:13.455552 27620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.486501 27620 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.4:44243
I20240503 14:13:13.486552 27746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.4:44243 every 8 connection(s)
I20240503 14:13:13.487823 27620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:13:13.491823 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27620
I20240503 14:13:13.492003 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:13:13.495671 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:0
--local_ip_for_outbound_sockets=127.20.99.5
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:13.501991 27747 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:13.502344 27747 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:13.503090 27747 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:13.504511 27169 ts_manager.cc:194] Registered new tserver with Master: c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243)
I20240503 14:13:13.505677 27169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.4:37845
I20240503 14:13:13.623461 27487 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
W20240503 14:13:13.828999 27750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:13.829298 27750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:13.829430 27750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:13.837064 27750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:13.837186 27750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:13.837347 27750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:13:13.849942 27750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:13.851377 27750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:13.853148 27750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:13.865489 27755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:13.866405 27758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:13.867094 27756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:13.867868 27750 server_base.cc:1034] running on GCE node
I20240503 14:13:13.868454 27750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:13.869370 27750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:13.870575 27750 hybrid_clock.cc:648] HybridClock initialized: now 1714745593870527 us; error 41 us; skew 500 ppm
I20240503 14:13:13.870877 27750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:13.873143 27750 webserver.cc:460] Webserver started at http://127.20.99.5:44565/ using document root <none> and password file <none>
I20240503 14:13:13.873829 27750 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:13.873942 27750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:13.874248 27750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:13.876767 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:13.877518 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:13.882553 27750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 14:13:13.886015 27764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.887423 27750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:13.887576 27750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:13.887776 27750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:13.909549 27750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:13.910497 27750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:13.910776 27750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:13.911789 27750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:13.913290 27750 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:13.913386 27750 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.913501 27750 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:13.913573 27750 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:13.945060 27750 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:37731
I20240503 14:13:13.945114 27876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:37731 every 8 connection(s)
I20240503 14:13:13.946393 27750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:13.956502 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27750
I20240503 14:13:13.956734 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:13:13.959394 27877 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:13.959678 27877 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:13.959972 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.6:0
--local_ip_for_outbound_sockets=127.20.99.6
--webserver_interface=127.20.99.6
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:13.960461 27877 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:13.961732 27170 ts_manager.cc:194] Registered new tserver with Master: d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731)
I20240503 14:13:13.962512 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:42721
I20240503 14:13:14.058593 27617 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
W20240503 14:13:14.285466 27880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:14.285763 27880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:14.285889 27880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:14.293448 27880 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:14.293579 27880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:14.293749 27880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.6
I20240503 14:13:14.302847 27880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.99.6
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.6
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:14.304250 27880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:14.306015 27880 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:14.318688 27885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:14.319540 27888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:14.320231 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:14.321121 27880 server_base.cc:1034] running on GCE node
I20240503 14:13:14.321756 27880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:14.322706 27880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:14.323894 27880 hybrid_clock.cc:648] HybridClock initialized: now 1714745594323844 us; error 54 us; skew 500 ppm
I20240503 14:13:14.324208 27880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:14.326390 27880 webserver.cc:460] Webserver started at http://127.20.99.6:46809/ using document root <none> and password file <none>
I20240503 14:13:14.327095 27880 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:14.327242 27880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:14.327567 27880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:14.330039 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:14.330790 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:14.335768 27880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240503 14:13:14.338930 27894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:14.340283 27880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:14.340436 27880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:14.340633 27880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:14.358688 27880 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:14.359537 27880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:14.359833 27880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:14.360823 27880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:14.362206 27880 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:14.362289 27880 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:14.362370 27880 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:14.362422 27880 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:14.393541 27880 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.6:45729
I20240503 14:13:14.393594 28006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.6:45729 every 8 connection(s)
I20240503 14:13:14.394826 27880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:13:14.397457 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 27880
I20240503 14:13:14.397632 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:13:14.400472 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.7:0
--local_ip_for_outbound_sockets=127.20.99.7
--webserver_interface=127.20.99.7
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:14.407732 28007 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:14.408094 28007 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:14.408854 28007 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:14.410231 27170 ts_manager.cc:194] Registered new tserver with Master: 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729)
I20240503 14:13:14.411072 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.6:50035
I20240503 14:13:14.508213 27747 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
W20240503 14:13:14.729120 28010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:14.729441 28010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:14.729588 28010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:14.737079 28010 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:14.737226 28010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:14.737393 28010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.7
I20240503 14:13:14.746870 28010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.99.7
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.7
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:14.748312 28010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:14.750016 28010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:14.762470 28015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:14.763447 28016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:14.764307 28018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:14.765107 28010 server_base.cc:1034] running on GCE node
I20240503 14:13:14.765738 28010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:14.766733 28010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:14.767941 28010 hybrid_clock.cc:648] HybridClock initialized: now 1714745594767893 us; error 46 us; skew 500 ppm
I20240503 14:13:14.768301 28010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:14.770599 28010 webserver.cc:460] Webserver started at http://127.20.99.7:46131/ using document root <none> and password file <none>
I20240503 14:13:14.771396 28010 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:14.771520 28010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:14.771826 28010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:13:14.774479 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:14.775352 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:14.780694 28010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240503 14:13:14.784469 28024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:14.785909 28010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:14.786084 28010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:14.786300 28010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:14.804822 28010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:14.805692 28010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:14.805970 28010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:14.806919 28010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:14.808908 28010 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:14.809006 28010 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:14.809123 28010 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:14.809192 28010 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:14.839128 28010 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.7:45199
I20240503 14:13:14.839236 28136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.7:45199 every 8 connection(s)
I20240503 14:13:14.840517 28010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:13:14.849900 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 28010
I20240503 14:13:14.850154 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:13:14.853377 28137 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:14.853674 28137 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:14.854389 28137 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:14.855770 27169 ts_manager.cc:194] Registered new tserver with Master: dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199)
I20240503 14:13:14.856568 27169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.7:41109
I20240503 14:13:14.866178 20876 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:13:14.881629 27169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44814:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:13:14.922135 27552 tablet_service.cc:1467] Processing CreateTablet for tablet 44eb2dd4af5444f2a6a75b50bce0e03f (DEFAULT_TABLE table=TestTable [id=a13bce5dfbf640a0a1c5187d91c1d4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.922134 27422 tablet_service.cc:1467] Processing CreateTablet for tablet 44eb2dd4af5444f2a6a75b50bce0e03f (DEFAULT_TABLE table=TestTable [id=a13bce5dfbf640a0a1c5187d91c1d4bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.923764 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eb2dd4af5444f2a6a75b50bce0e03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.923763 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eb2dd4af5444f2a6a75b50bce0e03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.937212 28153 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:14.938459 28154 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:14.941161 28153 tablet_bootstrap.cc:654] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.941779 28154 tablet_bootstrap.cc:654] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.942430 28153 log.cc:824] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:14.942930 28154 log.cc:824] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:14.945324 28153 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:14.945634 28154 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:14.945657 28153 ts_tablet_manager.cc:1367] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240503 14:13:14.945932 28154 ts_tablet_manager.cc:1367] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:13:14.952186 28153 raft_consensus.cc:348] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.952519 28153 raft_consensus.cc:374] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.952610 28153 raft_consensus.cc:729] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.953186 28153 consensus_queue.cc:260] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.953445 28154 raft_consensus.cc:348] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.953840 28154 raft_consensus.cc:374] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.953938 28154 raft_consensus.cc:729] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.954461 28154 consensus_queue.cc:260] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.954713 28153 ts_tablet_manager.cc:1398] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240503 14:13:14.955893 28154 ts_tablet_manager.cc:1398] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240503 14:13:14.964999 27877 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
I20240503 14:13:14.967824 28158 raft_consensus.cc:482] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:14.968008 28158 raft_consensus.cc:504] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.969326 28158 leader_election.cc:288] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:14.975400 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44eb2dd4af5444f2a6a75b50bce0e03f" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" is_pre_election: true
I20240503 14:13:14.975785 27572 raft_consensus.cc:2454] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:14.976291 27378 leader_election.cc:302] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:14.976608 28158 raft_consensus.cc:2781] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:14.976754 28158 raft_consensus.cc:482] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:14.976841 28158 raft_consensus.cc:3037] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.978719 28158 raft_consensus.cc:504] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.979132 28158 leader_election.cc:288] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:14.979586 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44eb2dd4af5444f2a6a75b50bce0e03f" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:14.979735 27572 raft_consensus.cc:3037] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.981670 27572 raft_consensus.cc:2454] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:14.982012 27378 leader_election.cc:302] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:14.982257 28158 raft_consensus.cc:2781] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:14.982635 28158 raft_consensus.cc:686] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:14.983019 28158 consensus_queue.cc:237] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:14.986645 27170 catalog_manager.cc:5484] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } }
I20240503 14:13:14.994812 20876 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:13:14.996769 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 60da896b8369489994355cc0e9050514 to finish bootstrapping
I20240503 14:13:15.005631 20876 ts_itest-base.cc:233] Waiting for 1 tablets on tserver aad738f0c10848789c2c811160990f8b to finish bootstrapping
I20240503 14:13:15.012905 20876 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2116de1ecc2f48daa3c8cca7edc82a95 to finish bootstrapping
I20240503 14:13:15.019625 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c57eaae6a75a43679120a10c6d3090c7 to finish bootstrapping
I20240503 14:13:15.028443 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d7a1a0df5dd047ddbacafd0870ed01d1 to finish bootstrapping
I20240503 14:13:15.036615 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 01c7323dc6f84fe0bc5eca4fcad8f078 to finish bootstrapping
W20240503 14:13:15.043345 27618 tablet.cc:2369] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:15.045820 20876 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dac9fd6f6a404989ae1a460ac3cde0fe to finish bootstrapping
I20240503 14:13:15.054720 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27620
I20240503 14:13:15.055825 27742 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:13:15.106166 27488 tablet.cc:2369] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:15.135846 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27620
I20240503 14:13:15.152376 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27750
I20240503 14:13:15.153496 27872 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:15.236146 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27750
I20240503 14:13:15.252444 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27880
I20240503 14:13:15.253618 28002 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:15.331971 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27880
I20240503 14:13:15.347685 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 28010
I20240503 14:13:15.348696 28132 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:13:15.373764 28158 consensus_queue.cc:1035] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.430202 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 28010
I20240503 14:13:19.197469 27170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44814:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:19.208313 27552 tablet_service.cc:1467] Processing CreateTablet for tablet 54fbf7658e87471895a21e486f59352a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.208313 27421 tablet_service.cc:1467] Processing CreateTablet for tablet b44e3d7641cc42fdb789fbfd0405f072 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.208505 27422 tablet_service.cc:1467] Processing CreateTablet for tablet f3923478811a4a0f82ec9f013ef7735d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.208828 27421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e3d7641cc42fdb789fbfd0405f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.208832 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fbf7658e87471895a21e486f59352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.209162 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3923478811a4a0f82ec9f013ef7735d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.215761 28185 tablet_bootstrap.cc:492] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:19.217957 28186 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:19.218997 28185 tablet_bootstrap.cc:654] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.219772 28186 tablet_bootstrap.cc:654] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.219810 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 54fbf7658e87471895a21e486f59352a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.220597 27289 tablet_service.cc:1467] Processing CreateTablet for tablet b44e3d7641cc42fdb789fbfd0405f072 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.221102 27291 tablet_service.cc:1467] Processing CreateTablet for tablet f3923478811a4a0f82ec9f013ef7735d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.221714 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fbf7658e87471895a21e486f59352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.222226 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3923478811a4a0f82ec9f013ef7735d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.222478 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e3d7641cc42fdb789fbfd0405f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.223078 28186 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:19.223229 28186 ts_tablet_manager.cc:1367] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 14:13:19.223827 28185 tablet_bootstrap.cc:492] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:19.223951 28185 ts_tablet_manager.cc:1367] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240503 14:13:19.224176 28186 raft_consensus.cc:348] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.224375 28186 raft_consensus.cc:374] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.224453 28186 raft_consensus.cc:729] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.224603 28185 raft_consensus.cc:348] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.224761 28185 raft_consensus.cc:374] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.224689 28186 consensus_queue.cc:260] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.224821 28185 raft_consensus.cc:729] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.225045 28185 consensus_queue.cc:260] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.225210 28186 ts_tablet_manager.cc:1398] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:19.225795 28185 ts_tablet_manager.cc:1398] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:19.226119 28186 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:19.229249 28186 tablet_bootstrap.cc:654] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.231537 28186 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:19.231696 28186 ts_tablet_manager.cc:1367] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:19.232422 28186 raft_consensus.cc:348] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.232651 28186 raft_consensus.cc:374] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.232738 28186 raft_consensus.cc:729] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.232971 28186 consensus_queue.cc:260] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.233963 28186 ts_tablet_manager.cc:1398] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:19.239485 28191 tablet_bootstrap.cc:492] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:19.242491 28191 tablet_bootstrap.cc:654] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.243564 28191 log.cc:824] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:19.245904 28191 tablet_bootstrap.cc:492] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:19.246125 28191 ts_tablet_manager.cc:1367] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:13:19.252348 28191 raft_consensus.cc:348] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.252647 28191 raft_consensus.cc:374] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.252719 28191 raft_consensus.cc:729] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.253230 28191 consensus_queue.cc:260] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.254627 28191 ts_tablet_manager.cc:1398] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240503 14:13:19.255023 28191 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:19.256479 28191 tablet_bootstrap.cc:654] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.258370 28191 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:19.258502 28191 ts_tablet_manager.cc:1367] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:19.259336 28191 raft_consensus.cc:348] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.259469 28191 raft_consensus.cc:374] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.259531 28191 raft_consensus.cc:729] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.259701 28191 consensus_queue.cc:260] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.260056 28191 ts_tablet_manager.cc:1398] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:19.260275 28191 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:19.261415 28191 tablet_bootstrap.cc:654] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.263135 28191 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:19.263291 28191 ts_tablet_manager.cc:1367] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:19.263763 28191 raft_consensus.cc:348] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.263913 28191 raft_consensus.cc:374] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.263984 28191 raft_consensus.cc:729] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.264183 28191 consensus_queue.cc:260] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.264626 28191 ts_tablet_manager.cc:1398] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:19.344825 28193 raft_consensus.cc:482] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.345104 28193 raft_consensus.cc:504] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.346645 28193 leader_election.cc:288] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:19.352926 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fbf7658e87471895a21e486f59352a" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" is_pre_election: true
I20240503 14:13:19.353145 27572 raft_consensus.cc:2454] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 0.
I20240503 14:13:19.353605 27247 leader_election.cc:302] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:19.353933 28193 raft_consensus.cc:2781] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.354066 28193 raft_consensus.cc:482] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.354182 28193 raft_consensus.cc:3037] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.356215 28193 raft_consensus.cc:504] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.356652 28193 leader_election.cc:288] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:19.357084 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fbf7658e87471895a21e486f59352a" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:19.357249 27572 raft_consensus.cc:3037] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.358974 27572 raft_consensus.cc:2454] T 54fbf7658e87471895a21e486f59352a P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 1.
I20240503 14:13:19.359336 27247 leader_election.cc:302] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:19.359567 28193 raft_consensus.cc:2781] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.359972 28193 raft_consensus.cc:686] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Leader. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:13:19.360395 28193 consensus_queue.cc:237] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:19.364030 27170 catalog_manager.cc:5484] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60da896b8369489994355cc0e9050514 (127.20.99.1). New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } }
W20240503 14:13:19.416210 27357 tablet.cc:2369] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:19.495620 28193 raft_consensus.cc:482] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.495949 28193 raft_consensus.cc:504] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.496578 28193 leader_election.cc:288] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:19.502408 27442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e3d7641cc42fdb789fbfd0405f072" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b" is_pre_election: true
I20240503 14:13:19.502768 27442 raft_consensus.cc:2454] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 0.
I20240503 14:13:19.503260 27245 leader_election.cc:302] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:19.503558 28193 raft_consensus.cc:2781] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.503666 28193 raft_consensus.cc:482] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.503746 28193 raft_consensus.cc:3037] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.505595 28193 raft_consensus.cc:504] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.506050 28193 leader_election.cc:288] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:19.506495 27442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e3d7641cc42fdb789fbfd0405f072" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:13:19.506654 27442 raft_consensus.cc:3037] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.508689 27442 raft_consensus.cc:2454] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 1.
I20240503 14:13:19.509011 27245 leader_election.cc:302] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:19.509243 28193 raft_consensus.cc:2781] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.509485 28193 raft_consensus.cc:686] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Leader. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:13:19.509775 28193 consensus_queue.cc:237] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.511962 27170 catalog_manager.cc:5484] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60da896b8369489994355cc0e9050514 (127.20.99.1). New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } }
I20240503 14:13:19.527583 28193 raft_consensus.cc:482] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.527760 28193 raft_consensus.cc:504] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.528270 28193 leader_election.cc:288] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:19.528828 27442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3923478811a4a0f82ec9f013ef7735d" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b" is_pre_election: true
I20240503 14:13:19.529124 27442 raft_consensus.cc:2454] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 0.
I20240503 14:13:19.529522 27245 leader_election.cc:302] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:19.529770 28193 raft_consensus.cc:2781] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.529881 28193 raft_consensus.cc:482] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.529948 28193 raft_consensus.cc:3037] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.531816 28193 raft_consensus.cc:504] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.532194 28193 leader_election.cc:288] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:19.532735 27442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3923478811a4a0f82ec9f013ef7735d" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:13:19.532914 27442 raft_consensus.cc:3037] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.534853 27442 raft_consensus.cc:2454] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 1.
I20240503 14:13:19.535264 27245 leader_election.cc:302] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:19.535553 28193 raft_consensus.cc:2781] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.535693 28193 raft_consensus.cc:686] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Leader. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:13:19.535962 28193 consensus_queue.cc:237] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:19.538259 27170 catalog_manager.cc:5484] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60da896b8369489994355cc0e9050514 (127.20.99.1). New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } }
I20240503 14:13:19.890110 28193 consensus_queue.cc:1035] T 54fbf7658e87471895a21e486f59352a P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.895888 28193 consensus_queue.cc:1035] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:19.910359 28203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:19.910667 28203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:19.918015 28203 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 14:13:19.997159 28193 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 196.072 ms
time per row: 0.0980361 ms
I20240503 14:13:20.259682 27164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44814:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:20.267849 27291 tablet_service.cc:1467] Processing CreateTablet for tablet 63dc99c47a8a4f379b3349837d4b7264 (DEFAULT_TABLE table=rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.268327 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63dc99c47a8a4f379b3349837d4b7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.268245 27551 tablet_service.cc:1467] Processing CreateTablet for tablet 244c146eb7624b3585068f9c2de0c4ed (DEFAULT_TABLE table=rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.268362 27552 tablet_service.cc:1467] Processing CreateTablet for tablet 63dc99c47a8a4f379b3349837d4b7264 (DEFAULT_TABLE table=rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.268754 27551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244c146eb7624b3585068f9c2de0c4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.269104 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63dc99c47a8a4f379b3349837d4b7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.269510 27422 tablet_service.cc:1467] Processing CreateTablet for tablet 244c146eb7624b3585068f9c2de0c4ed (DEFAULT_TABLE table=rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.269883 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244c146eb7624b3585068f9c2de0c4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.273720 28253 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:20.273656 27290 tablet_service.cc:1467] Processing CreateTablet for tablet a352cdeffa9a4b85badcefa545885bc3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.274031 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a352cdeffa9a4b85badcefa545885bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.275898 27550 tablet_service.cc:1467] Processing CreateTablet for tablet a352cdeffa9a4b85badcefa545885bc3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.276417 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a352cdeffa9a4b85badcefa545885bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.280566 28253 tablet_bootstrap.cc:654] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.283880 28254 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:20.284341 28253 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:20.284483 28253 ts_tablet_manager.cc:1367] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:13:20.285288 28253 raft_consensus.cc:348] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.285516 28253 raft_consensus.cc:374] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.285614 28253 raft_consensus.cc:729] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.285856 28253 consensus_queue.cc:260] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.286037 28254 tablet_bootstrap.cc:654] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.286306 28256 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:20.286494 28253 ts_tablet_manager.cc:1398] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:20.286733 28253 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:20.288244 28253 tablet_bootstrap.cc:654] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.289129 28256 tablet_bootstrap.cc:654] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.289320 28254 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:20.289505 28254 ts_tablet_manager.cc:1367] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 14:13:20.290658 28254 raft_consensus.cc:348] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.290918 28254 raft_consensus.cc:374] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.291075 28254 raft_consensus.cc:729] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.291574 28254 consensus_queue.cc:260] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.292483 28254 ts_tablet_manager.cc:1398] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:20.292894 28254 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:20.294984 28254 tablet_bootstrap.cc:654] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.297525 28256 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:20.297679 28256 ts_tablet_manager.cc:1367] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 14:13:20.298092 28254 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:20.298084 28253 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:20.298243 28253 ts_tablet_manager.cc:1367] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240503 14:13:20.298243 28254 ts_tablet_manager.cc:1367] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:20.298323 28256 raft_consensus.cc:348] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.298535 28256 raft_consensus.cc:374] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.298631 28256 raft_consensus.cc:729] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.298861 28256 consensus_queue.cc:260] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.299021 28253 raft_consensus.cc:348] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.299010 28254 raft_consensus.cc:348] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.299248 28254 raft_consensus.cc:374] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.299243 28253 raft_consensus.cc:374] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.299336 28254 raft_consensus.cc:729] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.299343 28256 ts_tablet_manager.cc:1398] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:20.299345 28253 raft_consensus.cc:729] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.299563 28253 consensus_queue.cc:260] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.299577 28254 consensus_queue.cc:260] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.300066 28253 ts_tablet_manager.cc:1398] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.301769 28254 ts_tablet_manager.cc:1398] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:20.302336 28253 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:20.304342 28253 tablet_bootstrap.cc:654] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.305833 28253 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:20.305960 28253 ts_tablet_manager.cc:1367] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:20.306486 28253 raft_consensus.cc:348] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.306636 28253 raft_consensus.cc:374] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.306731 28253 raft_consensus.cc:729] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.306944 28253 consensus_queue.cc:260] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.307402 28253 ts_tablet_manager.cc:1398] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:13:20.389936 28204 raft_consensus.cc:482] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.390134 28204 raft_consensus.cc:504] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.391588 28204 leader_election.cc:288] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:20.397398 27441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244c146eb7624b3585068f9c2de0c4ed" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b" is_pre_election: true
I20240503 14:13:20.397616 27441 raft_consensus.cc:2454] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 0.
I20240503 14:13:20.398098 27506 leader_election.cc:302] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:20.398444 28204 raft_consensus.cc:2781] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.398566 28204 raft_consensus.cc:482] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.398672 28204 raft_consensus.cc:3037] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.400980 28204 raft_consensus.cc:504] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.401257 28264 raft_consensus.cc:482] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.401410 28264 raft_consensus.cc:504] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.401537 28204 leader_election.cc:288] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:20.402006 27441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244c146eb7624b3585068f9c2de0c4ed" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:13:20.402104 28264 leader_election.cc:288] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:20.402215 27441 raft_consensus.cc:3037] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.404222 27441 raft_consensus.cc:2454] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 1.
I20240503 14:13:20.404580 27506 leader_election.cc:302] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:20.404830 28264 raft_consensus.cc:2781] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.405040 28264 raft_consensus.cc:686] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Becoming Leader. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:13:20.405666 28264 consensus_queue.cc:237] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:20.408316 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a352cdeffa9a4b85badcefa545885bc3" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514" is_pre_election: true
I20240503 14:13:20.408778 27311 raft_consensus.cc:2454] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 0.
I20240503 14:13:20.408869 27166 catalog_manager.cc:5484] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } }
I20240503 14:13:20.409299 27505 leader_election.cc:302] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:20.409570 28264 raft_consensus.cc:2781] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.409675 28264 raft_consensus.cc:482] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.409781 28264 raft_consensus.cc:3037] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.412091 28264 raft_consensus.cc:504] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.412504 28264 leader_election.cc:288] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:20.412935 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a352cdeffa9a4b85badcefa545885bc3" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:20.413146 27311 raft_consensus.cc:3037] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.415055 27311 raft_consensus.cc:2454] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 1.
I20240503 14:13:20.415462 27505 leader_election.cc:302] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:20.415737 28264 raft_consensus.cc:2781] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.415910 28264 raft_consensus.cc:686] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Becoming Leader. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:13:20.416153 28264 consensus_queue.cc:237] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:20.418187 27166 catalog_manager.cc:5484] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } }
I20240503 14:13:20.460309 28264 raft_consensus.cc:482] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.460517 28264 raft_consensus.cc:504] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.461094 28264 leader_election.cc:288] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:20.461696 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63dc99c47a8a4f379b3349837d4b7264" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514" is_pre_election: true
I20240503 14:13:20.461933 27311 raft_consensus.cc:2454] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 0.
I20240503 14:13:20.462333 27505 leader_election.cc:302] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:20.462600 28264 raft_consensus.cc:2781] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.462721 28264 raft_consensus.cc:482] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.462806 28264 raft_consensus.cc:3037] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.464637 28264 raft_consensus.cc:504] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.465041 28264 leader_election.cc:288] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:20.465507 27311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63dc99c47a8a4f379b3349837d4b7264" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:20.465685 27311 raft_consensus.cc:3037] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.467831 27311 raft_consensus.cc:2454] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 1.
I20240503 14:13:20.468175 27505 leader_election.cc:302] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:20.468477 28264 raft_consensus.cc:2781] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.468617 28264 raft_consensus.cc:686] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Becoming Leader. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:13:20.468870 28264 consensus_queue.cc:237] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:20.471354 27166 catalog_manager.cc:5484] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } }
W20240503 14:13:20.844007 28268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:20.844312 28268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:20.851758 28268 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:13:20.951560 28264 consensus_queue.cc:1035] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.981042 27311 raft_consensus.cc:1261] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:20.985174 28301 consensus_queue.cc:1035] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.993383 27311 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:20.995595 28301 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.022910 28297 mvcc.cc:205] Tried to move back new op lower bound from 7023597981591031808 to 7023597979306061824. Current Snapshot: MvccSnapshot[applied={T|T < 7023597981591031808}]
I20240503 14:13:21.054311 28306 mvcc.cc:205] Tried to move back new op lower bound from 7023597981591031808 to 7023597979306061824. Current Snapshot: MvccSnapshot[applied={T|T < 7023597981591031808}]
INSERT report
rows total: 2000
time total: 186.945 ms
time per row: 0.0934726 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:22.111045 27164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44814:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:22.120091 27291 tablet_service.cc:1467] Processing CreateTablet for tablet ae02587deb4a450da3171f30a1501950 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.120436 27421 tablet_service.cc:1467] Processing CreateTablet for tablet 86890b3c77ba41d7b08579defb20aff1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.120499 27422 tablet_service.cc:1467] Processing CreateTablet for tablet ae02587deb4a450da3171f30a1501950 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.120613 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae02587deb4a450da3171f30a1501950. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.120900 27421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86890b3c77ba41d7b08579defb20aff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.121237 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae02587deb4a450da3171f30a1501950. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.121201 27290 tablet_service.cc:1467] Processing CreateTablet for tablet dd18d7c1f6f64ef08d227e039e369cbc (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.121572 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd18d7c1f6f64ef08d227e039e369cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.121747 27420 tablet_service.cc:1467] Processing CreateTablet for tablet dd18d7c1f6f64ef08d227e039e369cbc (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.122093 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd18d7c1f6f64ef08d227e039e369cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.123911 27550 tablet_service.cc:1467] Processing CreateTablet for tablet 86890b3c77ba41d7b08579defb20aff1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.124310 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86890b3c77ba41d7b08579defb20aff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.133472 28325 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:22.135705 28325 tablet_bootstrap.cc:654] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.136816 28326 tablet_bootstrap.cc:492] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:22.138310 28327 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:22.138929 28326 tablet_bootstrap.cc:654] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.139801 28327 tablet_bootstrap.cc:654] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.140239 28325 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:22.140385 28325 ts_tablet_manager.cc:1367] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 14:13:22.140834 28326 tablet_bootstrap.cc:492] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:22.140996 28326 ts_tablet_manager.cc:1367] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 14:13:22.141192 28325 raft_consensus.cc:348] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.141410 28325 raft_consensus.cc:374] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.141520 28325 raft_consensus.cc:729] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.141788 28326 raft_consensus.cc:348] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.141844 28325 consensus_queue.cc:260] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.141983 28326 raft_consensus.cc:374] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.142102 28326 raft_consensus.cc:729] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.142383 28325 ts_tablet_manager.cc:1398] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:22.142338 28326 consensus_queue.cc:260] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.142712 28325 tablet_bootstrap.cc:492] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:22.142917 28326 ts_tablet_manager.cc:1398] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:22.143277 28326 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:22.144537 28325 tablet_bootstrap.cc:654] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.144778 28326 tablet_bootstrap.cc:654] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.148236 28327 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:22.148389 28327 ts_tablet_manager.cc:1367] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:13:22.149065 28327 raft_consensus.cc:348] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.149252 28327 raft_consensus.cc:374] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.149377 28327 raft_consensus.cc:729] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.149760 28327 consensus_queue.cc:260] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.151046 28326 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:22.151230 28325 tablet_bootstrap.cc:492] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:22.151291 28326 ts_tablet_manager.cc:1367] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240503 14:13:22.151427 28327 ts_tablet_manager.cc:1398] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:22.151480 28325 ts_tablet_manager.cc:1367] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 14:13:22.152112 28325 raft_consensus.cc:348] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.152288 28325 raft_consensus.cc:374] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.152357 28325 raft_consensus.cc:729] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.152603 28325 consensus_queue.cc:260] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.152923 28326 raft_consensus.cc:348] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.153205 28326 raft_consensus.cc:374] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.154171 28325 ts_tablet_manager.cc:1398] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:22.154198 28326 raft_consensus.cc:729] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.154533 28326 consensus_queue.cc:260] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.155010 28326 ts_tablet_manager.cc:1398] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:22.155306 28326 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:22.156980 28326 tablet_bootstrap.cc:654] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.158849 28326 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:22.159135 28326 ts_tablet_manager.cc:1367] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:22.159735 28326 raft_consensus.cc:348] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.159869 28326 raft_consensus.cc:374] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.159972 28326 raft_consensus.cc:729] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.160216 28326 consensus_queue.cc:260] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.160636 28326 ts_tablet_manager.cc:1398] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:22.177493 28272 raft_consensus.cc:482] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.177654 28272 raft_consensus.cc:504] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.178242 28272 leader_election.cc:288] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:22.183990 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514" is_pre_election: true
I20240503 14:13:22.184182 27310 raft_consensus.cc:2454] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:22.184574 27375 leader_election.cc:302] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:22.184816 28272 raft_consensus.cc:2781] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.184927 28272 raft_consensus.cc:482] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.184993 28272 raft_consensus.cc:3037] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.186785 28272 raft_consensus.cc:504] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.187222 28272 leader_election.cc:288] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:22.187680 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:22.187826 27310 raft_consensus.cc:3037] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.189719 27310 raft_consensus.cc:2454] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:22.190048 27375 leader_election.cc:302] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:22.190299 28272 raft_consensus.cc:2781] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.190662 28272 raft_consensus.cc:686] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:22.190901 28272 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:22.193151 27164 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } }
I20240503 14:13:22.371884 28303 raft_consensus.cc:482] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.372087 28303 raft_consensus.cc:504] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.372664 28303 leader_election.cc:288] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:22.373207 27441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86890b3c77ba41d7b08579defb20aff1" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b" is_pre_election: true
I20240503 14:13:22.373401 27441 raft_consensus.cc:2454] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 0.
I20240503 14:13:22.373829 27506 leader_election.cc:302] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:22.374102 28303 raft_consensus.cc:2781] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.374239 28303 raft_consensus.cc:482] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.374329 28303 raft_consensus.cc:3037] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.376185 28303 raft_consensus.cc:504] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.376596 28303 leader_election.cc:288] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:22.377048 27441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86890b3c77ba41d7b08579defb20aff1" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:13:22.377249 27441 raft_consensus.cc:3037] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.379148 27441 raft_consensus.cc:2454] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 1.
I20240503 14:13:22.379504 27506 leader_election.cc:302] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:22.379789 28303 raft_consensus.cc:2781] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.380144 28303 raft_consensus.cc:686] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Becoming Leader. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:13:22.380399 28303 consensus_queue.cc:237] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:22.382481 27164 catalog_manager.cc:5484] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } }
I20240503 14:13:22.460408 28272 raft_consensus.cc:482] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.460616 28272 raft_consensus.cc:504] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.461071 28272 leader_election.cc:288] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:22.461683 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae02587deb4a450da3171f30a1501950" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514" is_pre_election: true
I20240503 14:13:22.461898 27310 raft_consensus.cc:2454] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:22.462291 27375 leader_election.cc:302] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:22.462594 28272 raft_consensus.cc:2781] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.462689 28272 raft_consensus.cc:482] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.462782 28272 raft_consensus.cc:3037] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.464669 28272 raft_consensus.cc:504] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.465111 28272 leader_election.cc:288] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:22.465554 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae02587deb4a450da3171f30a1501950" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:22.465755 27310 raft_consensus.cc:3037] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.467542 27310 raft_consensus.cc:2454] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:22.467912 27375 leader_election.cc:302] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:22.468206 28272 raft_consensus.cc:2781] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.468354 28272 raft_consensus.cc:686] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:22.468597 28272 consensus_queue.cc:237] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } }
I20240503 14:13:22.470940 27164 catalog_manager.cc:5484] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } }
I20240503 14:13:22.749715 28272 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:22.820755 28341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:22.821095 28341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:22.828490 28341 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:13:22.921789 28303 consensus_queue.cc:1035] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.971642 27310 raft_consensus.cc:1261] T ae02587deb4a450da3171f30a1501950 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:22.972334 28336 consensus_queue.cc:1035] T ae02587deb4a450da3171f30a1501950 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.986500 28377 mvcc.cc:205] Tried to move back new op lower bound from 7023597989764653056 to 7023597987712761856. Current Snapshot: MvccSnapshot[applied={T|T < 7023597989764653056}]
INSERT report
rows total: 2000
time total: 172.711 ms
time per row: 0.0863554 ms
I20240503 14:13:23.151252 27168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44814:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:23.159456 27550 tablet_service.cc:1467] Processing CreateTablet for tablet f576c9fe2ddf4494b51d41d64d18a1eb (DEFAULT_TABLE table=rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.159770 27291 tablet_service.cc:1467] Processing CreateTablet for tablet f576c9fe2ddf4494b51d41d64d18a1eb (DEFAULT_TABLE table=rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.159919 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f576c9fe2ddf4494b51d41d64d18a1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.160241 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f576c9fe2ddf4494b51d41d64d18a1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.160540 27422 tablet_service.cc:1467] Processing CreateTablet for tablet 98bec1722172403aa21b3d060ed636d2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.160934 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98bec1722172403aa21b3d060ed636d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.163952 27290 tablet_service.cc:1467] Processing CreateTablet for tablet c723f0969bd9472d833bdbb54ee67556 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.164340 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c723f0969bd9472d833bdbb54ee67556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.167219 27551 tablet_service.cc:1467] Processing CreateTablet for tablet 98bec1722172403aa21b3d060ed636d2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.167371 27552 tablet_service.cc:1467] Processing CreateTablet for tablet c723f0969bd9472d833bdbb54ee67556 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.167692 27551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98bec1722172403aa21b3d060ed636d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.171034 28394 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:23.171388 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c723f0969bd9472d833bdbb54ee67556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.172976 28394 tablet_bootstrap.cc:654] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.185782 28397 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:23.189072 28397 tablet_bootstrap.cc:654] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.190668 28395 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:23.192528 28395 tablet_bootstrap.cc:654] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.193009 28394 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:23.193215 28394 ts_tablet_manager.cc:1367] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.005s
I20240503 14:13:23.194016 28394 raft_consensus.cc:348] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.194227 28394 raft_consensus.cc:374] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.194383 28397 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:23.194398 28394 raft_consensus.cc:729] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.194573 28397 ts_tablet_manager.cc:1367] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:23.194761 28394 consensus_queue.cc:260] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.195117 28395 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:23.195324 28395 ts_tablet_manager.cc:1367] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:23.195408 28397 raft_consensus.cc:348] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.195669 28397 raft_consensus.cc:374] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.195817 28397 raft_consensus.cc:729] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.196192 28397 consensus_queue.cc:260] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.196259 28395 raft_consensus.cc:348] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.196529 28395 raft_consensus.cc:374] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.196682 28395 raft_consensus.cc:729] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.196893 28397 ts_tablet_manager.cc:1398] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:23.196933 28395 consensus_queue.cc:260] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.197536 28395 ts_tablet_manager.cc:1398] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:23.197988 28395 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:23.200428 28394 ts_tablet_manager.cc:1398] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:23.200723 28394 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:23.201964 28395 tablet_bootstrap.cc:654] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.202150 28394 tablet_bootstrap.cc:654] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.206638 28394 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:23.206801 28394 ts_tablet_manager.cc:1367] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20240503 14:13:23.207630 28395 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:23.207592 28394 raft_consensus.cc:348] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.207777 28395 ts_tablet_manager.cc:1367] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240503 14:13:23.207799 28394 raft_consensus.cc:374] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.207867 28394 raft_consensus.cc:729] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.208098 28394 consensus_queue.cc:260] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.208524 28395 raft_consensus.cc:348] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.208660 28394 ts_tablet_manager.cc:1398] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:23.208724 28395 raft_consensus.cc:374] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.208849 28395 raft_consensus.cc:729] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.208926 28394 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:23.209131 28395 consensus_queue.cc:260] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.209650 28395 ts_tablet_manager.cc:1398] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:23.210449 28394 tablet_bootstrap.cc:654] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.212812 28394 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:23.212918 28394 ts_tablet_manager.cc:1367] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:23.213510 28394 raft_consensus.cc:348] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.213693 28394 raft_consensus.cc:374] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.213829 28394 raft_consensus.cc:729] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.214066 28394 consensus_queue.cc:260] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.214486 28394 ts_tablet_manager.cc:1398] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:23.249925 28193 raft_consensus.cc:482] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.250125 28193 raft_consensus.cc:504] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.250640 28193 leader_election.cc:288] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:23.251260 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c723f0969bd9472d833bdbb54ee67556" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" is_pre_election: true
I20240503 14:13:23.251498 27571 raft_consensus.cc:2454] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 0.
I20240503 14:13:23.251902 27247 leader_election.cc:302] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:23.252159 28193 raft_consensus.cc:2781] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.252301 28193 raft_consensus.cc:482] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.252389 28193 raft_consensus.cc:3037] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.254354 28193 raft_consensus.cc:504] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.254832 28193 leader_election.cc:288] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:23.255323 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c723f0969bd9472d833bdbb54ee67556" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:23.255474 27571 raft_consensus.cc:3037] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.257282 27571 raft_consensus.cc:2454] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 1.
I20240503 14:13:23.257634 27247 leader_election.cc:302] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:23.257879 28193 raft_consensus.cc:2781] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.258258 28193 raft_consensus.cc:686] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Leader. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:13:23.258771 28193 consensus_queue.cc:237] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.261090 27162 catalog_manager.cc:5484] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60da896b8369489994355cc0e9050514 (127.20.99.1). New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } }
I20240503 14:13:23.349987 28272 raft_consensus.cc:482] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.350184 28272 raft_consensus.cc:504] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.350670 28272 leader_election.cc:288] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:23.351173 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98bec1722172403aa21b3d060ed636d2" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" is_pre_election: true
I20240503 14:13:23.351401 27571 raft_consensus.cc:2454] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:23.351785 27378 leader_election.cc:302] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:23.352162 28272 raft_consensus.cc:2781] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.352238 28272 raft_consensus.cc:482] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.352344 28272 raft_consensus.cc:3037] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.354125 28272 raft_consensus.cc:504] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.354526 28272 leader_election.cc:288] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:23.354956 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98bec1722172403aa21b3d060ed636d2" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:23.355142 27571 raft_consensus.cc:3037] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.356921 27571 raft_consensus.cc:2454] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:23.357306 27378 leader_election.cc:302] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:23.357529 28272 raft_consensus.cc:2781] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.357677 28272 raft_consensus.cc:686] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:23.357909 28272 consensus_queue.cc:237] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:23.360234 27162 catalog_manager.cc:5484] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } }
I20240503 14:13:23.476997 28193 raft_consensus.cc:482] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.477200 28193 raft_consensus.cc:504] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.477720 28193 leader_election.cc:288] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:23.478300 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f576c9fe2ddf4494b51d41d64d18a1eb" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" is_pre_election: true
I20240503 14:13:23.478514 27571 raft_consensus.cc:2454] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 0.
I20240503 14:13:23.478863 27247 leader_election.cc:302] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:23.479131 28193 raft_consensus.cc:2781] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.479240 28193 raft_consensus.cc:482] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.479321 28193 raft_consensus.cc:3037] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.481154 28193 raft_consensus.cc:504] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.481560 28193 leader_election.cc:288] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:23.482038 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f576c9fe2ddf4494b51d41d64d18a1eb" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:23.482201 27571 raft_consensus.cc:3037] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.483949 27571 raft_consensus.cc:2454] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 1.
I20240503 14:13:23.484299 27247 leader_election.cc:302] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, 60da896b8369489994355cc0e9050514; no voters:
I20240503 14:13:23.484519 28193 raft_consensus.cc:2781] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.484658 28193 raft_consensus.cc:686] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Leader. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:13:23.484915 28193 consensus_queue.cc:237] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:23.487473 27162 catalog_manager.cc:5484] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60da896b8369489994355cc0e9050514 (127.20.99.1). New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } }
I20240503 14:13:23.775871 28193 consensus_queue.cc:1035] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:23.877677 28407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:23.878027 28407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:23.885386 28407 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:13:23.952081 28272 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:23.989130 28193 consensus_queue.cc:1035] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 183.037 ms
time per row: 0.0915183 ms
I20240503 14:13:24.210052 27168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44814:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:24.218747 27422 tablet_service.cc:1467] Processing CreateTablet for tablet 2c65982ba3034daeacd3407a5b6877e1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.218881 27291 tablet_service.cc:1467] Processing CreateTablet for tablet 2c65982ba3034daeacd3407a5b6877e1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.219267 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c65982ba3034daeacd3407a5b6877e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.219362 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c65982ba3034daeacd3407a5b6877e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.219817 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 4180d30f1d3449ff9c28c69599098ff4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.219887 27420 tablet_service.cc:1467] Processing CreateTablet for tablet 4180d30f1d3449ff9c28c69599098ff4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.220162 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4180d30f1d3449ff9c28c69599098ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.220243 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4180d30f1d3449ff9c28c69599098ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.223615 27421 tablet_service.cc:1467] Processing CreateTablet for tablet e8102c1d08074ea7bb5be6c5645d1fcd (DEFAULT_TABLE table=rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.223963 27421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8102c1d08074ea7bb5be6c5645d1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.227777 28462 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:24.228106 28461 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:24.229846 28462 tablet_bootstrap.cc:654] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.229952 28461 tablet_bootstrap.cc:654] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.229982 27552 tablet_service.cc:1467] Processing CreateTablet for tablet e8102c1d08074ea7bb5be6c5645d1fcd (DEFAULT_TABLE table=rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.230410 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8102c1d08074ea7bb5be6c5645d1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.235826 28462 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:24.235978 28462 ts_tablet_manager.cc:1367] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 14:13:24.236088 28461 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:24.236227 28461 ts_tablet_manager.cc:1367] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 14:13:24.236861 28465 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Bootstrap starting.
I20240503 14:13:24.236760 28462 raft_consensus.cc:348] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.236984 28462 raft_consensus.cc:374] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.237085 28462 raft_consensus.cc:729] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.237342 28462 consensus_queue.cc:260] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.237377 28461 raft_consensus.cc:348] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.237573 28461 raft_consensus.cc:374] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.237692 28461 raft_consensus.cc:729] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.237962 28461 consensus_queue.cc:260] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.238528 28462 ts_tablet_manager.cc:1398] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:24.238782 28462 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514: Bootstrap starting.
I20240503 14:13:24.238956 28465 tablet_bootstrap.cc:654] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.239872 28461 ts_tablet_manager.cc:1398] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:24.240139 28461 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:24.240451 28462 tablet_bootstrap.cc:654] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.241689 28461 tablet_bootstrap.cc:654] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.246591 28462 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514: No bootstrap required, opened a new log
I20240503 14:13:24.246773 28462 ts_tablet_manager.cc:1367] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:13:24.247895 28462 raft_consensus.cc:348] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.248185 28462 raft_consensus.cc:374] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.248273 28462 raft_consensus.cc:729] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.248585 28462 consensus_queue.cc:260] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.249890 28461 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:24.249869 28462 ts_tablet_manager.cc:1398] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:24.250002 28465 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: No bootstrap required, opened a new log
I20240503 14:13:24.250039 28461 ts_tablet_manager.cc:1367] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:13:24.250161 28465 ts_tablet_manager.cc:1367] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240503 14:13:24.250663 28461 raft_consensus.cc:348] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.250849 28461 raft_consensus.cc:374] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.250921 28461 raft_consensus.cc:729] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.251031 28465 raft_consensus.cc:348] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.251256 28465 raft_consensus.cc:374] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.251206 28461 consensus_queue.cc:260] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.251364 28465 raft_consensus.cc:729] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.251598 28465 consensus_queue.cc:260] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.251734 28461 ts_tablet_manager.cc:1398] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:24.252102 28461 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Bootstrap starting.
I20240503 14:13:24.252255 28465 ts_tablet_manager.cc:1398] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:24.253883 28461 tablet_bootstrap.cc:654] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.255924 28461 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: No bootstrap required, opened a new log
I20240503 14:13:24.256084 28461 ts_tablet_manager.cc:1367] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:13:24.256831 28461 raft_consensus.cc:348] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.257011 28461 raft_consensus.cc:374] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.257109 28461 raft_consensus.cc:729] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.257325 28461 consensus_queue.cc:260] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.257745 28461 ts_tablet_manager.cc:1398] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:24.282601 28272 raft_consensus.cc:482] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:24.282783 28272 raft_consensus.cc:504] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.283295 28272 leader_election.cc:288] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:24.283777 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c65982ba3034daeacd3407a5b6877e1" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514" is_pre_election: true
I20240503 14:13:24.283995 27310 raft_consensus.cc:2454] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:24.284410 27375 leader_election.cc:302] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:24.284684 28272 raft_consensus.cc:2781] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:24.284796 28272 raft_consensus.cc:482] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:24.284935 28272 raft_consensus.cc:3037] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.286803 28272 raft_consensus.cc:504] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.287379 28272 leader_election.cc:288] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:24.287839 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c65982ba3034daeacd3407a5b6877e1" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:24.288061 27310 raft_consensus.cc:3037] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.289891 27310 raft_consensus.cc:2454] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:24.290334 27375 leader_election.cc:302] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:24.290583 28272 raft_consensus.cc:2781] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:24.290741 28272 raft_consensus.cc:686] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:24.291043 28272 consensus_queue.cc:237] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.293581 27168 catalog_manager.cc:5484] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } }
I20240503 14:13:24.345994 28272 raft_consensus.cc:482] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:24.346174 28272 raft_consensus.cc:504] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.346691 28272 leader_election.cc:288] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:24.347314 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" is_pre_election: true
I20240503 14:13:24.347589 27571 raft_consensus.cc:2454] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:24.347982 27378 leader_election.cc:302] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:24.348239 28272 raft_consensus.cc:2781] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:24.348369 28272 raft_consensus.cc:482] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:24.348461 28272 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.350577 28272 raft_consensus.cc:504] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.351073 28272 leader_election.cc:288] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:24.351538 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:24.351737 27571 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.353601 27571 raft_consensus.cc:2454] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:24.353976 27378 leader_election.cc:302] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:24.354283 28272 raft_consensus.cc:2781] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:24.354413 28272 raft_consensus.cc:686] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:24.354650 28272 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } }
I20240503 14:13:24.357229 27168 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } }
I20240503 14:13:24.684113 28272 raft_consensus.cc:482] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:24.684330 28272 raft_consensus.cc:504] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.684861 28272 leader_election.cc:288] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:24.685618 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4180d30f1d3449ff9c28c69599098ff4" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514" is_pre_election: true
I20240503 14:13:24.685865 27310 raft_consensus.cc:2454] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 0.
I20240503 14:13:24.686266 27375 leader_election.cc:302] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:24.686545 28272 raft_consensus.cc:2781] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:24.686642 28272 raft_consensus.cc:482] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:24.686746 28272 raft_consensus.cc:3037] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.688627 28272 raft_consensus.cc:504] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.689026 28272 leader_election.cc:288] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:24.689481 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4180d30f1d3449ff9c28c69599098ff4" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:24.689646 27310 raft_consensus.cc:3037] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.691463 27310 raft_consensus.cc:2454] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 1.
I20240503 14:13:24.691841 27375 leader_election.cc:302] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:24.692088 28272 raft_consensus.cc:2781] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:24.692245 28272 raft_consensus.cc:686] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:24.692497 28272 consensus_queue.cc:237] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } }
I20240503 14:13:24.694813 27168 catalog_manager.cc:5484] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } }
I20240503 14:13:24.760274 28272 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.784940 28434 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:25.116449 28272 consensus_queue.cc:1035] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:25.125514 28478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:25.125890 28478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:25.133412 28478 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 177.867 ms
time per row: 0.0889336 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:26.386994 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.4:44243
--local_ip_for_outbound_sockets=127.20.99.4
--tserver_master_addrs=127.20.99.62:38501
--webserver_port=41603
--webserver_interface=127.20.99.4
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:13:26.706709 28522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:26.707020 28522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:26.707157 28522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:26.714721 28522 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:26.714887 28522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:26.715049 28522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.4
I20240503 14:13:26.724211 28522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.4:44243
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.99.4
--webserver_port=41603
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.4
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:26.725595 28522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:26.727392 28522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:26.739847 28528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:26.740787 28527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:26.741587 28530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:26.742379 28522 server_base.cc:1034] running on GCE node
I20240503 14:13:26.742970 28522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:26.743925 28522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:26.745118 28522 hybrid_clock.cc:648] HybridClock initialized: now 1714745606745081 us; error 47 us; skew 500 ppm
I20240503 14:13:26.745456 28522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:26.747637 28522 webserver.cc:460] Webserver started at http://127.20.99.4:41603/ using document root <none> and password file <none>
I20240503 14:13:26.748356 28522 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:26.748476 28522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:26.753643 28522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 14:13:26.756255 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:26.757485 28522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:26.757645 28522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c57eaae6a75a43679120a10c6d3090c7"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:26.758469 28522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:26.787405 28522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:26.788313 28522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:26.788619 28522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:26.789634 28522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:26.791138 28522 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:26.791239 28522 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:26.791374 28522 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:26.791437 28522 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:26.822882 28522 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.4:44243
I20240503 14:13:26.822940 28648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.4:44243 every 8 connection(s)
I20240503 14:13:26.824324 28522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:13:26.834420 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 28522
I20240503 14:13:26.835143 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:37731
--local_ip_for_outbound_sockets=127.20.99.5
--tserver_master_addrs=127.20.99.62:38501
--webserver_port=44565
--webserver_interface=127.20.99.5
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:26.836853 28649 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:26.837193 28649 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:26.838124 28649 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:26.839922 27170 ts_manager.cc:194] Re-registered known tserver with Master: c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243)
I20240503 14:13:26.841125 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.4:39679
W20240503 14:13:27.158309 28653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:27.158571 28653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:27.158664 28653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:27.166841 28653 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:27.166960 28653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:27.167120 28653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:13:27.176020 28653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:37731
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=44565
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:27.177263 28653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:27.178900 28653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:27.191037 28658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:27.191915 28659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:27.192765 28661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:27.193528 28653 server_base.cc:1034] running on GCE node
I20240503 14:13:27.194043 28653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:27.194908 28653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:27.196094 28653 hybrid_clock.cc:648] HybridClock initialized: now 1714745607196046 us; error 39 us; skew 500 ppm
I20240503 14:13:27.196386 28653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:27.198447 28653 webserver.cc:460] Webserver started at http://127.20.99.5:44565/ using document root <none> and password file <none>
I20240503 14:13:27.199101 28653 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:27.199234 28653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:27.204035 28653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:13:27.206511 28667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:27.207751 28653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:13:27.207895 28653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
format_stamp: "Formatted at 2024-05-03 14:13:13 on dist-test-slave-p0gh"
I20240503 14:13:27.208631 28653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:27.237519 28653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:27.238360 28653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:27.238621 28653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:27.239639 28653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:27.241356 28653 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:27.241446 28653 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:27.241572 28653 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:27.241639 28653 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:27.272933 28653 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:37731
I20240503 14:13:27.272987 28779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:37731 every 8 connection(s)
I20240503 14:13:27.274367 28653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:27.284188 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 28653
I20240503 14:13:27.284863 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.6:45729
--local_ip_for_outbound_sockets=127.20.99.6
--tserver_master_addrs=127.20.99.62:38501
--webserver_port=46809
--webserver_interface=127.20.99.6
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:27.286772 28780 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:27.287082 28780 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:27.287993 28780 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:27.289492 27170 ts_manager.cc:194] Re-registered known tserver with Master: d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731)
I20240503 14:13:27.290381 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:47621
W20240503 14:13:27.606268 28783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:27.606568 28783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:27.606693 28783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:27.614249 28783 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:27.614379 28783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:27.614547 28783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.6
I20240503 14:13:27.623463 28783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.6:45729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.99.6
--webserver_port=46809
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.6
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:27.624769 28783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:27.626446 28783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:27.638482 28788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:27.638998 28789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:27.639746 28791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:27.640437 28783 server_base.cc:1034] running on GCE node
I20240503 14:13:27.641007 28783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:27.641891 28783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:27.643113 28783 hybrid_clock.cc:648] HybridClock initialized: now 1714745607643063 us; error 36 us; skew 500 ppm
I20240503 14:13:27.643509 28783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:27.645711 28783 webserver.cc:460] Webserver started at http://127.20.99.6:46809/ using document root <none> and password file <none>
I20240503 14:13:27.646453 28783 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:27.646574 28783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:27.651671 28783 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:27.655073 28797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:27.656654 28783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:27.656811 28783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:27.657835 28783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:27.673134 28783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:27.673823 28783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:27.674084 28783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:27.674952 28783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:27.676359 28783 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:27.676429 28783 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:27.676535 28783 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:27.676582 28783 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:27.707160 28783 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.6:45729
I20240503 14:13:27.707219 28909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.6:45729 every 8 connection(s)
I20240503 14:13:27.708556 28783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:13:27.713037 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 28783
I20240503 14:13:27.713614 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.7:45199
--local_ip_for_outbound_sockets=127.20.99.7
--tserver_master_addrs=127.20.99.62:38501
--webserver_port=46131
--webserver_interface=127.20.99.7
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:27.721982 28910 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:27.722347 28910 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:27.723256 28910 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:27.724747 27170 ts_manager.cc:194] Re-registered known tserver with Master: 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729)
I20240503 14:13:27.725944 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.6:35143
I20240503 14:13:27.843555 28649 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
W20240503 14:13:28.041988 28913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:28.042304 28913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:28.042415 28913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:28.052068 28913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:28.052232 28913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:28.052484 28913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.7
I20240503 14:13:28.062361 28913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.7:45199
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.99.7
--webserver_port=46131
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.7
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:28.063771 28913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:28.065522 28913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:28.078459 28918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:28.079054 28919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:28.079963 28921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:28.080899 28913 server_base.cc:1034] running on GCE node
I20240503 14:13:28.081454 28913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:28.082376 28913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:28.083570 28913 hybrid_clock.cc:648] HybridClock initialized: now 1714745608083521 us; error 48 us; skew 500 ppm
I20240503 14:13:28.083894 28913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:28.086416 28913 webserver.cc:460] Webserver started at http://127.20.99.7:46131/ using document root <none> and password file <none>
I20240503 14:13:28.087121 28913 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:28.087289 28913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:28.092306 28913 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:28.094870 28927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:28.096169 28913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:28.096341 28913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
format_stamp: "Formatted at 2024-05-03 14:13:14 on dist-test-slave-p0gh"
I20240503 14:13:28.097121 28913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:28.120872 28913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:28.121699 28913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:28.122005 28913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:28.122941 28913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:28.124470 28913 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:28.124545 28913 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:28.124883 28913 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:28.124949 28913 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:28.159494 28913 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.7:45199
I20240503 14:13:28.159570 29039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.7:45199 every 8 connection(s)
I20240503 14:13:28.160889 28913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:13:28.163690 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 28913
I20240503 14:13:28.164033 20876 test_util.cc:268] Using random seed: 1390072449
I20240503 14:13:28.173398 29040 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:13:28.173660 29040 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:28.174307 29040 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:13:28.175504 27170 ts_manager.cc:194] Re-registered known tserver with Master: dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199)
I20240503 14:13:28.176277 27170 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.7:35933
I20240503 14:13:28.191841 20876 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 14:13:28.192270 20876 test_util.cc:268] Using random seed: 1390100713
I20240503 14:13:28.206631 20876 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 14:13:28.207005 20876 test_util.cc:268] Using random seed: 1390115426
I20240503 14:13:28.225021 20876 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 14:13:28.239112 20876 test_util.cc:268] Using random seed: 1390147537
I20240503 14:13:28.292721 28780 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
I20240503 14:13:28.311950 20876 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 14:13:28.312420 20876 test_util.cc:268] Using random seed: 1390220844
I20240503 14:13:28.421295 20876 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 14:13:28.729072 28910 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
I20240503 14:13:29.178330 29040 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
I20240503 14:13:29.852933 27291 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:29.870949 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:29.914368 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:29.921777 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:29.933142 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:29.946508 28975 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:30.076097 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:30.965142 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:30.981386 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:31.015326 27291 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:31.051098 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:31.098445 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:31.149096 28975 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:31.213126 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:31.635936 27440 consensus_queue.cc:237] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:31.645901 27310 raft_consensus.cc:1261] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20240503 14:13:31.646956 29140 consensus_queue.cc:1035] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20240503 14:13:31.650482 29140 raft_consensus.cc:2932] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:31.651641 27311 raft_consensus.cc:2932] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:31.667768 27166 catalog_manager.cc:5484] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 reported cstate change: config changed from index -1 to 61, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
W20240503 14:13:31.695685 27376 consensus_peers.cc:485] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b -> Peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729): Couldn't send request to peer 01c7323dc6f84fe0bc5eca4fcad8f078. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4180d30f1d3449ff9c28c69599098ff4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:31.968722 27570 consensus_queue.cc:237] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:31.984365 27440 raft_consensus.cc:1261] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240503 14:13:31.989732 29174 consensus_queue.cc:1035] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240503 14:13:31.992533 29174 raft_consensus.cc:2932] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index -1 to 91, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:31.997423 27440 raft_consensus.cc:2932] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index -1 to 91, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:32.005923 27170 catalog_manager.cc:5484] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 91, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
W20240503 14:13:32.030238 27505 consensus_peers.cc:485] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 -> Peer d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731): Couldn't send request to peer d7a1a0df5dd047ddbacafd0870ed01d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 244c146eb7624b3585068f9c2de0c4ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:32.259410 29303 ts_tablet_manager.cc:902] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:32.257025 27311 consensus_queue.cc:237] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:32.278146 27442 raft_consensus.cc:1261] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 109. (index mismatch)
I20240503 14:13:32.282439 29067 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240503 14:13:32.290122 29141 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:32.290743 27440 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:32.298156 27170 catalog_manager.cc:5484] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 108, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
W20240503 14:13:32.356741 27246 consensus_peers.cc:485] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3923478811a4a0f82ec9f013ef7735d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:32.399683 29303 tablet_copy_client.cc:308] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:13:32.420751 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet 4180d30f1d3449ff9c28c69599098ff4 from peer 01c7323dc6f84fe0bc5eca4fcad8f078 ({username='slave'} at 127.20.99.6:36947)
I20240503 14:13:32.421026 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet 4180d30f1d3449ff9c28c69599098ff4 from peer 01c7323dc6f84fe0bc5eca4fcad8f078 at {username='slave'} at 127.20.99.6:36947: session id = 01c7323dc6f84fe0bc5eca4fcad8f078-4180d30f1d3449ff9c28c69599098ff4
I20240503 14:13:32.428288 27462 tablet_copy_source_session.cc:205] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.447841 29303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4180d30f1d3449ff9c28c69599098ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.556000 29324 ts_tablet_manager.cc:902] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Initiating tablet copy from peer 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:32.668010 29303 tablet_copy_client.cc:783] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:32.668443 29303 tablet_copy_client.cc:647] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:32.678872 29303 tablet_copy_client.cc:523] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:32.687875 27442 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:32.691771 29303 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap starting.
I20240503 14:13:32.731662 29324 tablet_copy_client.cc:308] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43621
I20240503 14:13:32.742887 27572 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 14:13:32.743760 29138 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 14:13:32.747383 29165 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:32.754940 27166 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 81, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:32.770042 27570 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:32.774196 29335 ts_tablet_manager.cc:902] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Initiating tablet copy from peer 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:32.814164 27592 tablet_copy_service.cc:140] P 2116de1ecc2f48daa3c8cca7edc82a95: Received BeginTabletCopySession request for tablet 244c146eb7624b3585068f9c2de0c4ed from peer d7a1a0df5dd047ddbacafd0870ed01d1 ({username='slave'} at 127.20.99.5:34277)
I20240503 14:13:32.814419 27592 tablet_copy_service.cc:161] P 2116de1ecc2f48daa3c8cca7edc82a95: Beginning new tablet copy session on tablet 244c146eb7624b3585068f9c2de0c4ed from peer d7a1a0df5dd047ddbacafd0870ed01d1 at {username='slave'} at 127.20.99.5:34277: session id = d7a1a0df5dd047ddbacafd0870ed01d1-244c146eb7624b3585068f9c2de0c4ed
I20240503 14:13:32.817287 27592 tablet_copy_source_session.cc:205] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.826138 29303 log.cc:824] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:32.829604 29335 tablet_copy_client.cc:308] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:45565
I20240503 14:13:32.831980 29324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244c146eb7624b3585068f9c2de0c4ed. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:13:32.852495 27375 consensus_peers.cc:485] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b -> Peer dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199): Couldn't send request to peer dac9fd6f6a404989ae1a460ac3cde0fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8102c1d08074ea7bb5be6c5645d1fcd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:32.855459 29324 tablet_copy_client.cc:783] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:32.855878 29324 tablet_copy_client.cc:647] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:32.858999 29324 tablet_copy_client.cc:523] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:32.864295 29324 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap starting.
I20240503 14:13:32.884346 27331 tablet_copy_service.cc:140] P 60da896b8369489994355cc0e9050514: Received BeginTabletCopySession request for tablet f3923478811a4a0f82ec9f013ef7735d from peer c57eaae6a75a43679120a10c6d3090c7 ({username='slave'} at 127.20.99.4:34063)
I20240503 14:13:32.884656 27331 tablet_copy_service.cc:161] P 60da896b8369489994355cc0e9050514: Beginning new tablet copy session on tablet f3923478811a4a0f82ec9f013ef7735d from peer c57eaae6a75a43679120a10c6d3090c7 at {username='slave'} at 127.20.99.4:34063: session id = c57eaae6a75a43679120a10c6d3090c7-f3923478811a4a0f82ec9f013ef7735d
I20240503 14:13:32.889427 27331 tablet_copy_source_session.cc:205] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.895252 29335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3923478811a4a0f82ec9f013ef7735d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.922955 29324 log.cc:824] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:32.932714 29335 tablet_copy_client.cc:783] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:32.933085 29335 tablet_copy_client.cc:647] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:32.946053 29335 tablet_copy_client.cc:523] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:32.953341 29335 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Bootstrap starting.
I20240503 14:13:33.047155 29335 log.cc:824] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:33.099668 27310 consensus_queue.cc:237] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:33.102733 27572 raft_consensus.cc:1261] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 14:13:33.103341 29111 consensus_queue.cc:1035] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 14:13:33.105703 29111 raft_consensus.cc:2932] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:33.110574 27570 raft_consensus.cc:2932] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:33.116039 27166 catalog_manager.cc:5484] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: config changed from index -1 to 104, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:33.251691 29324 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=112 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.255991 29324 tablet_bootstrap.cc:492] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap complete.
I20240503 14:13:33.257408 29324 ts_tablet_manager.cc:1367] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent bootstrapping tablet: real 0.393s user 0.201s sys 0.002s
I20240503 14:13:33.267866 29372 ts_tablet_manager.cc:902] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:33.280154 29303 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.280959 29303 tablet_bootstrap.cc:492] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap complete.
I20240503 14:13:33.281138 29324 raft_consensus.cc:348] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:33.281675 29303 ts_tablet_manager.cc:1367] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent bootstrapping tablet: real 0.596s user 0.148s sys 0.018s
I20240503 14:13:33.281776 29324 raft_consensus.cc:729] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a1a0df5dd047ddbacafd0870ed01d1, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.289790 29303 raft_consensus.cc:348] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:33.290763 29303 raft_consensus.cc:729] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c7323dc6f84fe0bc5eca4fcad8f078, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.291657 29303 consensus_queue.cc:260] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:33.292589 29372 tablet_copy_client.cc:308] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:13:33.293032 29324 consensus_queue.cc:260] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 112, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:33.310663 29303 ts_tablet_manager.cc:1398] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent starting tablet: real 0.029s user 0.016s sys 0.000s
I20240503 14:13:33.314253 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet e8102c1d08074ea7bb5be6c5645d1fcd from peer dac9fd6f6a404989ae1a460ac3cde0fe ({username='slave'} at 127.20.99.7:34077)
I20240503 14:13:33.314502 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet e8102c1d08074ea7bb5be6c5645d1fcd from peer dac9fd6f6a404989ae1a460ac3cde0fe at {username='slave'} at 127.20.99.7:34077: session id = dac9fd6f6a404989ae1a460ac3cde0fe-e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:13:33.317456 27462 tablet_copy_source_session.cc:205] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:33.323088 29372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8102c1d08074ea7bb5be6c5645d1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.323025 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-4180d30f1d3449ff9c28c69599098ff4 received from {username='slave'} at 127.20.99.6:36947
I20240503 14:13:33.323227 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-4180d30f1d3449ff9c28c69599098ff4 on tablet 4180d30f1d3449ff9c28c69599098ff4 with peer 01c7323dc6f84fe0bc5eca4fcad8f078
I20240503 14:13:33.339803 29372 tablet_copy_client.cc:783] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:33.340337 29372 tablet_copy_client.cc:647] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:33.342962 29372 tablet_copy_client.cc:523] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:33.348088 29372 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap starting.
I20240503 14:13:33.373071 29335 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=125 overwritten=0 applied=125 ignored=0} inserts{seen=810 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.373752 29335 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Bootstrap complete.
I20240503 14:13:33.374303 29335 ts_tablet_manager.cc:1367] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Time spent bootstrapping tablet: real 0.421s user 0.182s sys 0.036s
I20240503 14:13:33.381273 29372 log.cc:824] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Log is configured to *not* fsync() on all Append() calls
W20240503 14:13:33.382099 27245 consensus_peers.cc:485] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 -> Peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729): Couldn't send request to peer 01c7323dc6f84fe0bc5eca4fcad8f078. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f576c9fe2ddf4494b51d41d64d18a1eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:33.403009 29335 raft_consensus.cc:348] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:33.404462 29335 raft_consensus.cc:729] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57eaae6a75a43679120a10c6d3090c7, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.407409 29335 consensus_queue.cc:260] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 125, Last appended: 1.125, Last appended by leader: 125, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:33.418404 29335 ts_tablet_manager.cc:1398] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Time spent starting tablet: real 0.044s user 0.011s sys 0.006s
I20240503 14:13:33.422358 27331 tablet_copy_service.cc:342] P 60da896b8369489994355cc0e9050514: Request end of tablet copy session c57eaae6a75a43679120a10c6d3090c7-f3923478811a4a0f82ec9f013ef7735d received from {username='slave'} at 127.20.99.4:34063
I20240503 14:13:33.422533 27331 tablet_copy_service.cc:433] P 60da896b8369489994355cc0e9050514: ending tablet copy session c57eaae6a75a43679120a10c6d3090c7-f3923478811a4a0f82ec9f013ef7735d on tablet f3923478811a4a0f82ec9f013ef7735d with peer c57eaae6a75a43679120a10c6d3090c7
I20240503 14:13:33.560854 29324 ts_tablet_manager.cc:1398] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent starting tablet: real 0.303s user 0.095s sys 0.101s
I20240503 14:13:33.562873 27592 tablet_copy_service.cc:342] P 2116de1ecc2f48daa3c8cca7edc82a95: Request end of tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-244c146eb7624b3585068f9c2de0c4ed received from {username='slave'} at 127.20.99.5:34277
I20240503 14:13:33.563068 27592 tablet_copy_service.cc:433] P 2116de1ecc2f48daa3c8cca7edc82a95: ending tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-244c146eb7624b3585068f9c2de0c4ed on tablet 244c146eb7624b3585068f9c2de0c4ed with peer d7a1a0df5dd047ddbacafd0870ed01d1
I20240503 14:13:33.609304 29303 ts_tablet_manager.cc:902] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: Initiating tablet copy from peer 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:33.659332 29303 tablet_copy_client.cc:308] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:45565
I20240503 14:13:33.659833 29372 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=717 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.662593 29372 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap complete.
I20240503 14:13:33.663156 29372 ts_tablet_manager.cc:1367] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent bootstrapping tablet: real 0.315s user 0.191s sys 0.019s
I20240503 14:13:33.668139 29372 raft_consensus.cc:348] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:33.668526 29372 raft_consensus.cc:729] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.669273 29372 consensus_queue.cc:260] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:33.671002 29372 ts_tablet_manager.cc:1398] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20240503 14:13:33.673184 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-e8102c1d08074ea7bb5be6c5645d1fcd received from {username='slave'} at 127.20.99.7:34077
I20240503 14:13:33.673329 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-e8102c1d08074ea7bb5be6c5645d1fcd on tablet e8102c1d08074ea7bb5be6c5645d1fcd with peer dac9fd6f6a404989ae1a460ac3cde0fe
I20240503 14:13:33.681368 27331 tablet_copy_service.cc:140] P 60da896b8369489994355cc0e9050514: Received BeginTabletCopySession request for tablet f576c9fe2ddf4494b51d41d64d18a1eb from peer 01c7323dc6f84fe0bc5eca4fcad8f078 ({username='slave'} at 127.20.99.6:58751)
I20240503 14:13:33.681586 27331 tablet_copy_service.cc:161] P 60da896b8369489994355cc0e9050514: Beginning new tablet copy session on tablet f576c9fe2ddf4494b51d41d64d18a1eb from peer 01c7323dc6f84fe0bc5eca4fcad8f078 at {username='slave'} at 127.20.99.6:58751: session id = 01c7323dc6f84fe0bc5eca4fcad8f078-f576c9fe2ddf4494b51d41d64d18a1eb
I20240503 14:13:33.686012 27331 tablet_copy_source_session.cc:205] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:33.687669 29303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f576c9fe2ddf4494b51d41d64d18a1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.696585 29303 tablet_copy_client.cc:783] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:33.697042 29303 tablet_copy_client.cc:647] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:33.702438 29303 tablet_copy_client.cc:523] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:33.710052 29303 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap starting.
I20240503 14:13:33.749239 28604 raft_consensus.cc:1203] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.107->[1.108-1.129] Dedup: 1.125->[1.126-1.129]
I20240503 14:13:33.825562 27442 consensus_queue.cc:237] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:33.828352 27570 raft_consensus.cc:1261] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240503 14:13:33.829005 29140 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240503 14:13:33.832363 29140 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:33.838135 27571 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:33.840931 27166 catalog_manager.cc:5484] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 105, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:33.860803 28865 raft_consensus.cc:1203] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.97] Dedup: 1.78->[1.79-1.97]
W20240503 14:13:33.874806 27375 consensus_peers.cc:485] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b -> Peer d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731): Couldn't send request to peer d7a1a0df5dd047ddbacafd0870ed01d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98bec1722172403aa21b3d060ed636d2. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:34.031060 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 51236.9us
I20240503 14:13:34.155340 29303 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:34.155839 29303 tablet_bootstrap.cc:492] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap complete.
I20240503 14:13:34.156206 29303 ts_tablet_manager.cc:1367] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent bootstrapping tablet: real 0.446s user 0.185s sys 0.058s
I20240503 14:13:34.157609 29303 raft_consensus.cc:348] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:34.157892 29303 raft_consensus.cc:729] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c7323dc6f84fe0bc5eca4fcad8f078, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.158241 29303 consensus_queue.cc:260] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:34.158988 29303 ts_tablet_manager.cc:1398] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:34.159670 27331 tablet_copy_service.cc:342] P 60da896b8369489994355cc0e9050514: Request end of tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-f576c9fe2ddf4494b51d41d64d18a1eb received from {username='slave'} at 127.20.99.6:58751
I20240503 14:13:34.159795 27331 tablet_copy_service.cc:433] P 60da896b8369489994355cc0e9050514: ending tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-f576c9fe2ddf4494b51d41d64d18a1eb on tablet f576c9fe2ddf4494b51d41d64d18a1eb with peer 01c7323dc6f84fe0bc5eca4fcad8f078
I20240503 14:13:34.166620 28735 raft_consensus.cc:1203] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.90->[1.91-1.132] Dedup: 1.112->[1.113-1.132]
I20240503 14:13:34.193608 28865 raft_consensus.cc:1203] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.103->[1.104-1.111] Dedup: 1.108->[1.109-1.111]
I20240503 14:13:34.257331 28995 raft_consensus.cc:1203] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.95] Dedup: 1.86->[1.87-1.95]
I20240503 14:13:34.421207 29448 ts_tablet_manager.cc:902] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:34.480489 29448 tablet_copy_client.cc:308] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:13:34.515414 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet 98bec1722172403aa21b3d060ed636d2 from peer d7a1a0df5dd047ddbacafd0870ed01d1 ({username='slave'} at 127.20.99.5:52387)
I20240503 14:13:34.515651 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet 98bec1722172403aa21b3d060ed636d2 from peer d7a1a0df5dd047ddbacafd0870ed01d1 at {username='slave'} at 127.20.99.5:52387: session id = d7a1a0df5dd047ddbacafd0870ed01d1-98bec1722172403aa21b3d060ed636d2
I20240503 14:13:34.522148 27462 tablet_copy_source_session.cc:205] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:34.525794 29448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98bec1722172403aa21b3d060ed636d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.587697 29448 tablet_copy_client.cc:783] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:34.588105 29448 tablet_copy_client.cc:647] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:34.594539 29448 tablet_copy_client.cc:523] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:34.641659 29448 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap starting.
I20240503 14:13:34.679111 27572 consensus_queue.cc:237] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 144, Committed index: 144, Last appended: 1.145, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:34.683372 27310 raft_consensus.cc:1261] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240503 14:13:34.683977 28303 consensus_queue.cc:1035] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240503 14:13:34.689141 28303 raft_consensus.cc:2932] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Committing config change with OpId 1.146: config changed from index -1 to 146, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:34.692883 27310 raft_consensus.cc:2932] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index -1 to 146, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:34.705538 27170 catalog_manager.cc:5484] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 reported cstate change: config changed from index -1 to 146, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
W20240503 14:13:34.733942 27507 consensus_peers.cc:485] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63dc99c47a8a4f379b3349837d4b7264. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:34.966514 29448 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=113 overwritten=0 applied=113 ignored=0} inserts{seen=744 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:34.967087 29448 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap complete.
I20240503 14:13:34.967454 29448 ts_tablet_manager.cc:1367] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent bootstrapping tablet: real 0.326s user 0.160s sys 0.017s
I20240503 14:13:34.968477 29448 raft_consensus.cc:348] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:34.968940 29448 raft_consensus.cc:729] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a1a0df5dd047ddbacafd0870ed01d1, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.969220 29448 consensus_queue.cc:260] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 113, Last appended: 1.113, Last appended by leader: 113, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:34.991497 29448 ts_tablet_manager.cc:1398] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent starting tablet: real 0.024s user 0.002s sys 0.015s
I20240503 14:13:34.992720 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-98bec1722172403aa21b3d060ed636d2 received from {username='slave'} at 127.20.99.5:52387
I20240503 14:13:34.992880 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-98bec1722172403aa21b3d060ed636d2 on tablet 98bec1722172403aa21b3d060ed636d2 with peer d7a1a0df5dd047ddbacafd0870ed01d1
I20240503 14:13:35.077765 27571 consensus_queue.cc:237] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 1.156, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:35.081419 27307 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 156. Preceding OpId from leader: term: 1 index: 157. (index mismatch)
I20240503 14:13:35.083022 28303 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240503 14:13:35.093137 28303 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Committing config change with OpId 1.157: config changed from index -1 to 157, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:35.107913 27307 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Committing config change with OpId 1.157: config changed from index -1 to 157, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:35.116298 27170 catalog_manager.cc:5484] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: config changed from index -1 to 157, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
W20240503 14:13:35.245569 27505 consensus_peers.cc:485] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 -> Peer dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199): Couldn't send request to peer dac9fd6f6a404989ae1a460ac3cde0fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a352cdeffa9a4b85badcefa545885bc3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:35.298430 29500 ts_tablet_manager.cc:902] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: Initiating tablet copy from peer 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:35.346006 29500 tablet_copy_client.cc:308] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43621
I20240503 14:13:35.376346 27592 tablet_copy_service.cc:140] P 2116de1ecc2f48daa3c8cca7edc82a95: Received BeginTabletCopySession request for tablet 63dc99c47a8a4f379b3349837d4b7264 from peer c57eaae6a75a43679120a10c6d3090c7 ({username='slave'} at 127.20.99.4:40891)
I20240503 14:13:35.376567 27592 tablet_copy_service.cc:161] P 2116de1ecc2f48daa3c8cca7edc82a95: Beginning new tablet copy session on tablet 63dc99c47a8a4f379b3349837d4b7264 from peer c57eaae6a75a43679120a10c6d3090c7 at {username='slave'} at 127.20.99.4:40891: session id = c57eaae6a75a43679120a10c6d3090c7-63dc99c47a8a4f379b3349837d4b7264
I20240503 14:13:35.389839 27592 tablet_copy_source_session.cc:205] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:35.396219 28735 raft_consensus.cc:1203] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.104->[1.105-1.129] Dedup: 1.113->[1.114-1.129]
I20240503 14:13:35.397920 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63dc99c47a8a4f379b3349837d4b7264. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.404359 29500 tablet_copy_client.cc:783] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:35.404633 29500 tablet_copy_client.cc:647] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:35.418565 29500 tablet_copy_client.cc:523] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:35.422106 29500 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap starting.
I20240503 14:13:35.445111 27569 consensus_queue.cc:237] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
W20240503 14:13:35.447780 27505 consensus_peers.cc:485] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 -> Peer d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731): Couldn't send request to peer d7a1a0df5dd047ddbacafd0870ed01d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86890b3c77ba41d7b08579defb20aff1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:35.453696 27442 raft_consensus.cc:1261] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240503 14:13:35.470115 29461 consensus_queue.cc:1035] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240503 14:13:35.486538 29464 raft_consensus.cc:2932] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index -1 to 99, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:35.499037 27170 catalog_manager.cc:5484] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: config changed from index -1 to 99, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:35.499979 27442 raft_consensus.cc:2932] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
W20240503 14:13:35.604295 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 47358.4us
I20240503 14:13:35.720324 29533 ts_tablet_manager.cc:902] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Initiating tablet copy from peer 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:35.811465 27442 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
W20240503 14:13:35.813724 27376 consensus_peers.cc:485] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b -> Peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729): Couldn't send request to peer 01c7323dc6f84fe0bc5eca4fcad8f078. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd18d7c1f6f64ef08d227e039e369cbc. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:35.828127 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:37731, user_credentials={real_user=slave}} blocked reactor thread for 70351.1us
I20240503 14:13:35.855459 29500 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=164 overwritten=0 applied=164 ignored=0} inserts{seen=849 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:35.855906 29500 tablet_bootstrap.cc:492] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap complete.
I20240503 14:13:35.856189 29500 ts_tablet_manager.cc:1367] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: Time spent bootstrapping tablet: real 0.434s user 0.213s sys 0.031s
I20240503 14:13:35.856949 29500 raft_consensus.cc:348] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:35.857142 29500 raft_consensus.cc:729] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57eaae6a75a43679120a10c6d3090c7, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.857412 29500 consensus_queue.cc:260] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:35.867763 29500 ts_tablet_manager.cc:1398] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20240503 14:13:35.868975 27592 tablet_copy_service.cc:342] P 2116de1ecc2f48daa3c8cca7edc82a95: Request end of tablet copy session c57eaae6a75a43679120a10c6d3090c7-63dc99c47a8a4f379b3349837d4b7264 received from {username='slave'} at 127.20.99.4:40891
I20240503 14:13:35.869316 27592 tablet_copy_service.cc:433] P 2116de1ecc2f48daa3c8cca7edc82a95: ending tablet copy session c57eaae6a75a43679120a10c6d3090c7-63dc99c47a8a4f379b3349837d4b7264 on tablet 63dc99c47a8a4f379b3349837d4b7264 with peer c57eaae6a75a43679120a10c6d3090c7
I20240503 14:13:35.888715 27307 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240503 14:13:35.899946 29420 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240503 14:13:35.906471 29165 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:35.907641 27307 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:35.915057 27170 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 104, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:35.924223 29533 tablet_copy_client.cc:308] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43621
I20240503 14:13:35.946416 27592 tablet_copy_service.cc:140] P 2116de1ecc2f48daa3c8cca7edc82a95: Received BeginTabletCopySession request for tablet a352cdeffa9a4b85badcefa545885bc3 from peer dac9fd6f6a404989ae1a460ac3cde0fe ({username='slave'} at 127.20.99.7:54125)
I20240503 14:13:35.946589 27592 tablet_copy_service.cc:161] P 2116de1ecc2f48daa3c8cca7edc82a95: Beginning new tablet copy session on tablet a352cdeffa9a4b85badcefa545885bc3 from peer dac9fd6f6a404989ae1a460ac3cde0fe at {username='slave'} at 127.20.99.7:54125: session id = dac9fd6f6a404989ae1a460ac3cde0fe-a352cdeffa9a4b85badcefa545885bc3
I20240503 14:13:35.951134 27592 tablet_copy_source_session.cc:205] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:35.954723 29533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a352cdeffa9a4b85badcefa545885bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.965260 29533 tablet_copy_client.cc:783] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:35.965718 29533 tablet_copy_client.cc:647] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:35.982281 29533 tablet_copy_client.cc:523] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:35.987306 29533 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap starting.
I20240503 14:13:36.073727 29557 ts_tablet_manager.cc:902] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: Initiating tablet copy from peer 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:13:36.099704 28603 raft_consensus.cc:1203] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.145->[1.146-1.178] Dedup: 1.164->[1.165-1.178]
I20240503 14:13:36.148569 29557 tablet_copy_client.cc:308] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43621
I20240503 14:13:36.149502 27592 tablet_copy_service.cc:140] P 2116de1ecc2f48daa3c8cca7edc82a95: Received BeginTabletCopySession request for tablet 86890b3c77ba41d7b08579defb20aff1 from peer d7a1a0df5dd047ddbacafd0870ed01d1 ({username='slave'} at 127.20.99.5:34277)
I20240503 14:13:36.149708 27592 tablet_copy_service.cc:161] P 2116de1ecc2f48daa3c8cca7edc82a95: Beginning new tablet copy session on tablet 86890b3c77ba41d7b08579defb20aff1 from peer d7a1a0df5dd047ddbacafd0870ed01d1 at {username='slave'} at 127.20.99.5:34277: session id = d7a1a0df5dd047ddbacafd0870ed01d1-86890b3c77ba41d7b08579defb20aff1
I20240503 14:13:36.152599 27592 tablet_copy_source_session.cc:205] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:36.156375 29557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86890b3c77ba41d7b08579defb20aff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.184405 29557 tablet_copy_client.cc:783] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:36.186630 29557 tablet_copy_client.cc:647] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:36.190105 29557 tablet_copy_client.cc:523] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:36.194763 29557 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap starting.
I20240503 14:13:36.371789 29575 ts_tablet_manager.cc:902] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:36.423003 29575 tablet_copy_client.cc:308] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:13:36.425621 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet dd18d7c1f6f64ef08d227e039e369cbc from peer 01c7323dc6f84fe0bc5eca4fcad8f078 ({username='slave'} at 127.20.99.6:36947)
I20240503 14:13:36.425856 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet dd18d7c1f6f64ef08d227e039e369cbc from peer 01c7323dc6f84fe0bc5eca4fcad8f078 at {username='slave'} at 127.20.99.6:36947: session id = 01c7323dc6f84fe0bc5eca4fcad8f078-dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:13:36.429555 27462 tablet_copy_source_session.cc:205] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:36.430913 29575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd18d7c1f6f64ef08d227e039e369cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.437811 29575 tablet_copy_client.cc:783] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:36.438123 29575 tablet_copy_client.cc:647] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:36.448596 29575 tablet_copy_client.cc:523] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:36.452893 29575 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap starting.
I20240503 14:13:36.457151 29557 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=110 overwritten=0 applied=109 ignored=0} inserts{seen=740 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:13:36.457739 29557 tablet_bootstrap.cc:492] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap complete.
I20240503 14:13:36.458091 29557 ts_tablet_manager.cc:1367] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent bootstrapping tablet: real 0.263s user 0.159s sys 0.029s
I20240503 14:13:36.458863 29557 raft_consensus.cc:348] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:36.459573 29557 raft_consensus.cc:729] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a1a0df5dd047ddbacafd0870ed01d1, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.459887 29557 consensus_queue.cc:260] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 109, Last appended: 1.110, Last appended by leader: 110, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:36.473207 29557 ts_tablet_manager.cc:1398] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent starting tablet: real 0.015s user 0.002s sys 0.006s
I20240503 14:13:36.480881 27592 tablet_copy_service.cc:342] P 2116de1ecc2f48daa3c8cca7edc82a95: Request end of tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-86890b3c77ba41d7b08579defb20aff1 received from {username='slave'} at 127.20.99.5:34277
I20240503 14:13:36.481074 27592 tablet_copy_service.cc:433] P 2116de1ecc2f48daa3c8cca7edc82a95: ending tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-86890b3c77ba41d7b08579defb20aff1 on tablet 86890b3c77ba41d7b08579defb20aff1 with peer d7a1a0df5dd047ddbacafd0870ed01d1
I20240503 14:13:36.497704 28734 raft_consensus.cc:1203] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.98->[1.99-1.111] Dedup: 1.110->[1.111-1.111]
W20240503 14:13:36.524334 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 116140us
I20240503 14:13:36.570891 29533 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=171 overwritten=0 applied=171 ignored=0} inserts{seen=843 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:36.571537 29533 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap complete.
I20240503 14:13:36.571911 29533 ts_tablet_manager.cc:1367] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent bootstrapping tablet: real 0.585s user 0.233s sys 0.017s
I20240503 14:13:36.572968 29533 raft_consensus.cc:348] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:36.573311 29533 raft_consensus.cc:729] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.573603 29533 consensus_queue.cc:260] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 171, Last appended: 1.171, Last appended by leader: 171, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:36.580751 29533 ts_tablet_manager.cc:1398] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:13:36.586047 27592 tablet_copy_service.cc:342] P 2116de1ecc2f48daa3c8cca7edc82a95: Request end of tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-a352cdeffa9a4b85badcefa545885bc3 received from {username='slave'} at 127.20.99.7:54125
I20240503 14:13:36.586190 27592 tablet_copy_service.cc:433] P 2116de1ecc2f48daa3c8cca7edc82a95: ending tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-a352cdeffa9a4b85badcefa545885bc3 on tablet a352cdeffa9a4b85badcefa545885bc3 with peer dac9fd6f6a404989ae1a460ac3cde0fe
W20240503 14:13:36.647672 27245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 67871us
I20240503 14:13:36.669632 29575 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=780 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:36.670047 29575 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap complete.
I20240503 14:13:36.670307 29575 ts_tablet_manager.cc:1367] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent bootstrapping tablet: real 0.218s user 0.162s sys 0.007s
I20240503 14:13:36.671101 29575 raft_consensus.cc:348] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:36.671346 29575 raft_consensus.cc:729] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c7323dc6f84fe0bc5eca4fcad8f078, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.671582 29575 consensus_queue.cc:260] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:36.733016 28995 raft_consensus.cc:1203] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.156->[1.157-1.181] Dedup: 1.171->[1.172-1.181]
I20240503 14:13:36.766580 29575 ts_tablet_manager.cc:1398] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent starting tablet: real 0.096s user 0.030s sys 0.058s
I20240503 14:13:36.768235 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-dd18d7c1f6f64ef08d227e039e369cbc received from {username='slave'} at 127.20.99.6:36947
I20240503 14:13:36.768373 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-dd18d7c1f6f64ef08d227e039e369cbc on tablet dd18d7c1f6f64ef08d227e039e369cbc with peer 01c7323dc6f84fe0bc5eca4fcad8f078
I20240503 14:13:36.800913 27311 consensus_queue.cc:237] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:36.803359 27568 raft_consensus.cc:1261] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240503 14:13:36.803450 28865 raft_consensus.cc:1203] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.103->[1.104-1.109] Dedup: 1.108->[1.109-1.109]
W20240503 14:13:36.803623 27246 consensus_peers.cc:485] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c723f0969bd9472d833bdbb54ee67556. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:36.803995 29404 consensus_queue.cc:1035] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20240503 14:13:36.806722 29589 raft_consensus.cc:2932] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index -1 to 121, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:36.811630 27568 raft_consensus.cc:2932] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index -1 to 121, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:36.831341 27170 catalog_manager.cc:5484] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 reported cstate change: config changed from index -1 to 121, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:37.249863 27311 consensus_queue.cc:237] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.174, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:37.279635 27442 raft_consensus.cc:1261] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240503 14:13:37.281915 29589 consensus_queue.cc:1035] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240503 14:13:37.290400 29067 raft_consensus.cc:2932] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Committing config change with OpId 1.175: config changed from index -1 to 175, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:37.294772 29620 ts_tablet_manager.cc:902] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: Initiating tablet copy from peer 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:37.298031 27166 catalog_manager.cc:5484] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 reported cstate change: config changed from index -1 to 175, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
W20240503 14:13:37.322707 27244 consensus_peers.cc:485] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 -> Peer dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199): Couldn't send request to peer dac9fd6f6a404989ae1a460ac3cde0fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b44e3d7641cc42fdb789fbfd0405f072. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.345638 27442 raft_consensus.cc:2932] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index -1 to 175, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:37.366252 29620 tablet_copy_client.cc:308] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:45565
I20240503 14:13:37.366950 27331 tablet_copy_service.cc:140] P 60da896b8369489994355cc0e9050514: Received BeginTabletCopySession request for tablet c723f0969bd9472d833bdbb54ee67556 from peer c57eaae6a75a43679120a10c6d3090c7 ({username='slave'} at 127.20.99.4:34063)
I20240503 14:13:37.367113 27331 tablet_copy_service.cc:161] P 60da896b8369489994355cc0e9050514: Beginning new tablet copy session on tablet c723f0969bd9472d833bdbb54ee67556 from peer c57eaae6a75a43679120a10c6d3090c7 at {username='slave'} at 127.20.99.4:34063: session id = c57eaae6a75a43679120a10c6d3090c7-c723f0969bd9472d833bdbb54ee67556
I20240503 14:13:37.370395 27331 tablet_copy_source_session.cc:205] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:37.373982 29620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c723f0969bd9472d833bdbb54ee67556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.380896 29620 tablet_copy_client.cc:783] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:37.381140 29620 tablet_copy_client.cc:647] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:37.385003 29620 tablet_copy_client.cc:523] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:37.388630 29620 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap starting.
I20240503 14:13:37.611552 27440 consensus_queue.cc:237] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:37.698954 29620 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=133 overwritten=0 applied=133 ignored=0} inserts{seen=805 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:37.700623 29620 tablet_bootstrap.cc:492] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap complete.
I20240503 14:13:37.701150 29620 ts_tablet_manager.cc:1367] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: Time spent bootstrapping tablet: real 0.313s user 0.226s sys 0.048s
I20240503 14:13:37.703351 29620 raft_consensus.cc:348] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:37.703789 29620 raft_consensus.cc:729] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57eaae6a75a43679120a10c6d3090c7, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.704211 29620 consensus_queue.cc:260] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 133, Last appended: 1.133, Last appended by leader: 133, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:37.712134 29620 ts_tablet_manager.cc:1398] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20240503 14:13:37.719489 27331 tablet_copy_service.cc:342] P 60da896b8369489994355cc0e9050514: Request end of tablet copy session c57eaae6a75a43679120a10c6d3090c7-c723f0969bd9472d833bdbb54ee67556 received from {username='slave'} at 127.20.99.4:34063
I20240503 14:13:37.719666 27331 tablet_copy_service.cc:433] P 60da896b8369489994355cc0e9050514: ending tablet copy session c57eaae6a75a43679120a10c6d3090c7-c723f0969bd9472d833bdbb54ee67556 on tablet c723f0969bd9472d833bdbb54ee67556 with peer c57eaae6a75a43679120a10c6d3090c7
I20240503 14:13:37.758239 28603 raft_consensus.cc:1203] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.120->[1.121-1.140] Dedup: 1.133->[1.134-1.140]
I20240503 14:13:37.784356 27568 raft_consensus.cc:1261] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:37.784320 27375 consensus_peers.cc:485] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b -> Peer dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199): Couldn't send request to peer dac9fd6f6a404989ae1a460ac3cde0fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44eb2dd4af5444f2a6a75b50bce0e03f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.784965 29165 consensus_queue.cc:1035] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:37.805954 29606 raft_consensus.cc:2932] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:37.808715 27572 raft_consensus.cc:2932] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:37.822151 27166 catalog_manager.cc:5484] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 2, VOTER dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:37.883714 29652 ts_tablet_manager.cc:902] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: Initiating tablet copy from peer 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:13:37.954604 29652 tablet_copy_client.cc:308] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:45565
I20240503 14:13:38.015648 27331 tablet_copy_service.cc:140] P 60da896b8369489994355cc0e9050514: Received BeginTabletCopySession request for tablet b44e3d7641cc42fdb789fbfd0405f072 from peer dac9fd6f6a404989ae1a460ac3cde0fe ({username='slave'} at 127.20.99.7:40945)
I20240503 14:13:38.015842 27331 tablet_copy_service.cc:161] P 60da896b8369489994355cc0e9050514: Beginning new tablet copy session on tablet b44e3d7641cc42fdb789fbfd0405f072 from peer dac9fd6f6a404989ae1a460ac3cde0fe at {username='slave'} at 127.20.99.7:40945: session id = dac9fd6f6a404989ae1a460ac3cde0fe-b44e3d7641cc42fdb789fbfd0405f072
I20240503 14:13:38.018719 27331 tablet_copy_source_session.cc:205] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.021695 29652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e3d7641cc42fdb789fbfd0405f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.030498 29652 tablet_copy_client.cc:783] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.030867 29652 tablet_copy_client.cc:647] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.039357 29652 tablet_copy_client.cc:523] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.044124 29652 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap starting.
W20240503 14:13:38.340106 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 61289.4us
I20240503 14:13:38.356397 29685 ts_tablet_manager.cc:902] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:38.404556 29652 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=185 overwritten=0 applied=184 ignored=0} inserts{seen=856 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:13:38.405377 29652 tablet_bootstrap.cc:492] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap complete.
I20240503 14:13:38.405867 29652 ts_tablet_manager.cc:1367] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent bootstrapping tablet: real 0.362s user 0.235s sys 0.028s
I20240503 14:13:38.407018 29652 raft_consensus.cc:348] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:38.408314 29652 raft_consensus.cc:729] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.408694 29652 consensus_queue.cc:260] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 184, Last appended: 1.185, Last appended by leader: 185, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:38.417016 29652 ts_tablet_manager.cc:1398] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent starting tablet: real 0.011s user 0.002s sys 0.006s
I20240503 14:13:38.419697 27331 tablet_copy_service.cc:342] P 60da896b8369489994355cc0e9050514: Request end of tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-b44e3d7641cc42fdb789fbfd0405f072 received from {username='slave'} at 127.20.99.7:40945
I20240503 14:13:38.419827 27331 tablet_copy_service.cc:433] P 60da896b8369489994355cc0e9050514: ending tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-b44e3d7641cc42fdb789fbfd0405f072 on tablet b44e3d7641cc42fdb789fbfd0405f072 with peer dac9fd6f6a404989ae1a460ac3cde0fe
I20240503 14:13:38.732187 29685 tablet_copy_client.cc:308] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:13:38.733457 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet 44eb2dd4af5444f2a6a75b50bce0e03f from peer dac9fd6f6a404989ae1a460ac3cde0fe ({username='slave'} at 127.20.99.7:34077)
I20240503 14:13:38.733649 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet 44eb2dd4af5444f2a6a75b50bce0e03f from peer dac9fd6f6a404989ae1a460ac3cde0fe at {username='slave'} at 127.20.99.7:34077: session id = dac9fd6f6a404989ae1a460ac3cde0fe-44eb2dd4af5444f2a6a75b50bce0e03f
I20240503 14:13:38.736704 27462 tablet_copy_source_session.cc:205] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.738941 29685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44eb2dd4af5444f2a6a75b50bce0e03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.745291 29685 tablet_copy_client.cc:783] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.750792 29685 tablet_copy_client.cc:647] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.753918 29685 tablet_copy_client.cc:523] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.757431 29685 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap starting.
I20240503 14:13:38.793793 29685 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.794447 29685 tablet_bootstrap.cc:492] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: Bootstrap complete.
I20240503 14:13:38.794775 29685 ts_tablet_manager.cc:1367] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20240503 14:13:38.797799 29685 raft_consensus.cc:348] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:38.798152 29685 raft_consensus.cc:729] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.798485 29685 consensus_queue.cc:260] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:38.805598 29685 ts_tablet_manager.cc:1398] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:38.812402 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-44eb2dd4af5444f2a6a75b50bce0e03f received from {username='slave'} at 127.20.99.7:34077
I20240503 14:13:38.812585 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session dac9fd6f6a404989ae1a460ac3cde0fe-44eb2dd4af5444f2a6a75b50bce0e03f on tablet 44eb2dd4af5444f2a6a75b50bce0e03f with peer dac9fd6f6a404989ae1a460ac3cde0fe
I20240503 14:13:38.939905 28994 raft_consensus.cc:1203] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.174->[1.175-1.194] Dedup: 1.185->[1.186-1.194]
I20240503 14:13:39.286918 28995 raft_consensus.cc:1203] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:39.312420 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:39.561913 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:39.625391 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:39.685024 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:39.761139 28975 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:39.781486 27291 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:39.787119 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:13:40.206936 27245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 47292.8us
I20240503 14:13:40.597359 27441 consensus_queue.cc:237] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:40.746882 28993 raft_consensus.cc:1261] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.747687 29728 consensus_queue.cc:1035] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.828356 29728 raft_consensus.cc:2932] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:40.831149 28993 raft_consensus.cc:2932] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:40.841375 27170 catalog_manager.cc:5484] T 44eb2dd4af5444f2a6a75b50bce0e03f P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 2 to 3, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:41.062299 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet 44eb2dd4af5444f2a6a75b50bce0e03f with delete_type TABLET_DATA_TOMBSTONED (TS 2116de1ecc2f48daa3c8cca7edc82a95 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34100
W20240503 14:13:41.167217 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 73217.5us
I20240503 14:13:41.183601 29795 tablet_replica.cc:330] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: stopping tablet replica
I20240503 14:13:41.188956 29795 raft_consensus.cc:2229] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.192041 29795 raft_consensus.cc:2258] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.249794 29795 ts_tablet_manager.cc:1875] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.258522 29795 ts_tablet_manager.cc:1888] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:13:41.258751 29795 log.cc:1196] T 44eb2dd4af5444f2a6a75b50bce0e03f P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/44eb2dd4af5444f2a6a75b50bce0e03f
I20240503 14:13:41.260177 27157 catalog_manager.cc:4830] TS 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): tablet 44eb2dd4af5444f2a6a75b50bce0e03f (table TestTable [id=a13bce5dfbf640a0a1c5187d91c1d4bc]) successfully deleted
W20240503 14:13:41.978663 27245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 83004.7us
I20240503 14:13:42.361169 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:42.371273 27291 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:42.380921 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:42.416733 28975 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:42.422947 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:42.442302 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:42.535660 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:13:42.843058 27244 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 48144.4us
I20240503 14:13:42.970603 27311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b44e3d7641cc42fdb789fbfd0405f072"
dest_uuid: "60da896b8369489994355cc0e9050514"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58798
I20240503 14:13:42.970911 27311 raft_consensus.cc:595] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:42.982030 29623 raft_consensus.cc:979] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: : Instructing follower dac9fd6f6a404989ae1a460ac3cde0fe to start an election
I20240503 14:13:42.983150 29828 raft_consensus.cc:1067] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Signalling peer dac9fd6f6a404989ae1a460ac3cde0fe to start an election
W20240503 14:13:42.989243 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:13:42.991819 28995 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b44e3d7641cc42fdb789fbfd0405f072"
dest_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
from {username='slave'} at 127.20.99.1:51951
W20240503 14:13:42.995463 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:13:42.997740 28995 raft_consensus.cc:482] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:42.997943 28995 raft_consensus.cc:3037] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:42.999265 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:13:43.004937 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:13:43.015481 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:13:43.021234 28995 raft_consensus.cc:504] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
W20240503 14:13:43.025023 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:13:43.037433 28995 leader_election.cc:288] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565), aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:13:43.038988 27442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e3d7641cc42fdb789fbfd0405f072" candidate_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 255 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:13:43.039266 27442 raft_consensus.cc:3037] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.042003 27442 raft_consensus.cc:2454] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac9fd6f6a404989ae1a460ac3cde0fe in term 2.
W20240503 14:13:43.045437 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:13:43.048112 27307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e3d7641cc42fdb789fbfd0405f072" candidate_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 255 } } ignore_live_leader: true dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:13:43.048426 27307 raft_consensus.cc:3032] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:43.048578 27307 raft_consensus.cc:729] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:13:43.048875 27307 consensus_queue.cc:260] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 1.255, Last appended by leader: 255, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:43.049283 27307 raft_consensus.cc:3037] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:43.054205 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Illegal state: replica 60da896b8369489994355cc0e9050514 is not leader of this config: current role FOLLOWER
I20240503 14:13:43.055549 27307 raft_consensus.cc:2454] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac9fd6f6a404989ae1a460ac3cde0fe in term 2.
I20240503 14:13:43.056174 28928 leader_election.cc:302] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, dac9fd6f6a404989ae1a460ac3cde0fe; no voters:
I20240503 14:13:43.076700 29875 raft_consensus.cc:2781] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:43.080936 29875 raft_consensus.cc:686] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 LEADER]: Becoming Leader. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Running, Role: LEADER
I20240503 14:13:43.101259 29875 consensus_queue.cc:237] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 255, Committed index: 255, Last appended: 1.255, Last appended by leader: 255, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:43.118106 27166 catalog_manager.cc:5484] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: term changed from 1 to 2, leader changed from 60da896b8369489994355cc0e9050514 (127.20.99.1) to dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7). New cstate: current_term: 2 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
W20240503 14:13:43.360606 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46712: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:13:43.439630 27307 raft_consensus.cc:1261] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20240503 14:13:43.441291 27442 raft_consensus.cc:1261] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 255. Preceding OpId from leader: term: 2 index: 257. (index mismatch)
I20240503 14:13:43.442019 29879 consensus_queue.cc:1035] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20240503 14:13:43.454802 29875 consensus_queue.cc:1035] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 256, Last known committed idx: 255, Time since last communication: 0.000s
I20240503 14:13:43.459115 29794 mvcc.cc:205] Tried to move back new op lower bound from 7023598073554829312 to 7023598072228532224. Current Snapshot: MvccSnapshot[applied={T|T < 7023598073554829312 or (T in {7023598073554829312})}]
I20240503 14:13:44.162809 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:44.198043 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:44.225539 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:44.233834 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:44.283147 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:44.302515 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:44.348623 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:13:45.212102 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 49301us
I20240503 14:13:45.418148 28992 consensus_queue.cc:237] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 2.292, Last appended by leader: 255, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:13:45.419811 27442 raft_consensus.cc:1261] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 292. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20240503 14:13:45.420399 29875 consensus_queue.cc:1035] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240503 14:13:45.422792 29875 raft_consensus.cc:2932] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 LEADER]: Committing config change with OpId 2.293: config changed from index 175 to 293, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:45.426460 27442 raft_consensus.cc:2932] T b44e3d7641cc42fdb789fbfd0405f072 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Committing config change with OpId 2.293: config changed from index 175 to 293, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:45.432199 27166 catalog_manager.cc:5484] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 175 to 293, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:13:45.478085 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet b44e3d7641cc42fdb789fbfd0405f072 with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 293) from {username='slave'} at 127.0.0.1:60240
I20240503 14:13:45.519842 29977 tablet_replica.cc:330] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:13:45.520608 29977 raft_consensus.cc:2229] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.521018 29977 raft_consensus.cc:2258] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.523483 29977 ts_tablet_manager.cc:1875] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:45.536417 29977 ts_tablet_manager.cc:1888] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.292
I20240503 14:13:45.536670 29977 log.cc:1196] T b44e3d7641cc42fdb789fbfd0405f072 P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/b44e3d7641cc42fdb789fbfd0405f072
I20240503 14:13:45.539299 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet b44e3d7641cc42fdb789fbfd0405f072 (table rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]) successfully deleted
I20240503 14:13:46.435006 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:46.475314 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:46.477406 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:46.516777 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:46.596843 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:46.606161 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:46.697782 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:47.489991 27307 consensus_queue.cc:237] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:47.510195 28603 raft_consensus.cc:1261] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240503 14:13:47.514017 30054 consensus_queue.cc:1035] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240503 14:13:47.543059 30054 raft_consensus.cc:2932] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index 121 to 267, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:47.548194 28603 raft_consensus.cc:2932] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index 121 to 267, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:47.559737 27170 catalog_manager.cc:5484] T c723f0969bd9472d833bdbb54ee67556 P 60da896b8369489994355cc0e9050514 reported cstate change: config changed from index 121 to 267, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:47.659971 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet c723f0969bd9472d833bdbb54ee67556 with delete_type TABLET_DATA_TOMBSTONED (TS 2116de1ecc2f48daa3c8cca7edc82a95 not found in new config with opid_index 267) from {username='slave'} at 127.0.0.1:34100
I20240503 14:13:47.674564 30074 tablet_replica.cc:330] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: stopping tablet replica
I20240503 14:13:47.674912 30074 raft_consensus.cc:2229] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.675664 30074 raft_consensus.cc:2258] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.677183 30074 ts_tablet_manager.cc:1875] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:47.683899 30074 ts_tablet_manager.cc:1888] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.266
I20240503 14:13:47.684103 30074 log.cc:1196] T c723f0969bd9472d833bdbb54ee67556 P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/c723f0969bd9472d833bdbb54ee67556
I20240503 14:13:47.686252 27157 catalog_manager.cc:4830] TS 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): tablet c723f0969bd9472d833bdbb54ee67556 (table rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]) successfully deleted
W20240503 14:13:48.305490 27506 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 56372.5us
W20240503 14:13:48.334374 27505 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 86848.6us
W20240503 14:13:48.392617 27505 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 56401.8us
W20240503 14:13:49.068017 27245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 86965.8us
W20240503 14:13:49.396589 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 49015.7us
I20240503 14:13:49.898584 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.969480 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.988947 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:49.992314 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:50.013417 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:50.120432 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:50.224571 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:51.351034 27442 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:13:51.354250 28864 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch)
I20240503 14:13:51.359334 30159 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s
I20240503 14:13:51.363349 30078 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 104 to 296, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:51.370302 27170 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 104 to 296, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:51.372359 28864 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 104 to 296, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:13:51.514237 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet dd18d7c1f6f64ef08d227e039e369cbc with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 296) from {username='slave'} at 127.0.0.1:60240
I20240503 14:13:51.520061 30213 tablet_replica.cc:330] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:13:51.535526 30213 raft_consensus.cc:2229] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.559957 30213 pending_rounds.cc:62] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Trying to abort 1 pending ops.
I20240503 14:13:51.560204 30213 pending_rounds.cc:68] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Aborting op as it isn't in flight: id { term: 1 index: 295 } timestamp: 7023598105954521088 op_type: WRITE_OP write_request { tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000d\220\221!\003\367\301\214\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""515fe86e5f292d56" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023598105671770112 authz_token { token_data: "\010\264\344\323\261\006\"3\n\005slave\022*\n 1a271df1692246a6aa89d544c5e294b2\020\001\030\001 \001(\001" signature: "<redacted>""j\237\226\316|\266*\030{\347\017(E\336\355\225\255\304)\003\224\365-\326\235\037\211\360\201_%P\'?\005\364T\013\220\226 \234\257\rzD\001f\225\035;\240\243\211H\007\2203\304\223\301e\317\233" signing_key_seq_num: 0 } } request_id { client_id: "5a18ca8952de4a72af2f3d4d0b8ca0a9" seq_no: 823 first_incomplete_seq_no: 823 attempt_no: 0 }
I20240503 14:13:51.561421 30213 raft_consensus.cc:2258] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.589169 30213 ts_tablet_manager.cc:1875] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:51.658855 30213 ts_tablet_manager.cc:1888] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.295
I20240503 14:13:51.664791 30213 log.cc:1196] T dd18d7c1f6f64ef08d227e039e369cbc P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:13:51.669485 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet dd18d7c1f6f64ef08d227e039e369cbc (table rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]) successfully deleted
I20240503 14:13:52.530675 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:52.545648 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:52.547128 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:52.585559 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:52.664079 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:52.751273 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:52.754477 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:13:53.322465 27245 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:45729, user_credentials={real_user=slave}} blocked reactor thread for 60718us
W20240503 14:13:53.410542 27505 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 51551us
I20240503 14:13:53.621423 27571 consensus_queue.cc:237] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:53.624722 28734 raft_consensus.cc:1261] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240503 14:13:53.625517 30181 consensus_queue.cc:1035] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240503 14:13:53.629181 30181 raft_consensus.cc:2932] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 99 to 293, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:53.632527 28734 raft_consensus.cc:2932] T 86890b3c77ba41d7b08579defb20aff1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 99 to 293, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:53.677946 27164 catalog_manager.cc:5484] T 86890b3c77ba41d7b08579defb20aff1 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: config changed from index 99 to 293, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:53.909991 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet 86890b3c77ba41d7b08579defb20aff1 with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 293) from {username='slave'} at 127.0.0.1:50958
I20240503 14:13:53.911738 30313 tablet_replica.cc:330] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:13:53.918411 30313 raft_consensus.cc:2229] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:53.930523 30313 raft_consensus.cc:2258] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:53.947901 30313 ts_tablet_manager.cc:1875] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:53.959031 30313 ts_tablet_manager.cc:1888] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.292
I20240503 14:13:53.965723 30313 log.cc:1196] T 86890b3c77ba41d7b08579defb20aff1 P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/86890b3c77ba41d7b08579defb20aff1
I20240503 14:13:53.968771 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet 86890b3c77ba41d7b08579defb20aff1 (table rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]) successfully deleted
W20240503 14:13:53.979964 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:37731, user_credentials={real_user=slave}} blocked reactor thread for 48815.5us
I20240503 14:13:55.386041 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.393575 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:55.434082 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.457845 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:55.509497 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:55.550808 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:13:55.645255 27376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:45729, user_credentials={real_user=slave}} blocked reactor thread for 51115.7us
I20240503 14:13:55.680140 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:56.453738 27572 consensus_queue.cc:237] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 582, Committed index: 582, Last appended: 1.582, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:13:56.460484 28604 raft_consensus.cc:1261] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 582. Preceding OpId from leader: term: 1 index: 583. (index mismatch)
I20240503 14:13:56.462396 30180 consensus_queue.cc:1035] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 583, Last known committed idx: 582, Time since last communication: 0.000s
I20240503 14:13:56.465988 30180 raft_consensus.cc:2932] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Committing config change with OpId 1.583: config changed from index 146 to 583, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:56.470636 28604 raft_consensus.cc:2932] T 63dc99c47a8a4f379b3349837d4b7264 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Committing config change with OpId 1.583: config changed from index 146 to 583, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:56.476751 27164 catalog_manager.cc:5484] T 63dc99c47a8a4f379b3349837d4b7264 P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: config changed from index 146 to 583, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:13:56.559897 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet 63dc99c47a8a4f379b3349837d4b7264 with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 583) from {username='slave'} at 127.0.0.1:60240
I20240503 14:13:56.568436 30409 tablet_replica.cc:330] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:13:56.568724 30409 raft_consensus.cc:2229] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.569022 30409 raft_consensus.cc:2258] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:56.574158 30409 ts_tablet_manager.cc:1875] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.630314 30409 ts_tablet_manager.cc:1888] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.582
I20240503 14:13:56.630445 30409 log.cc:1196] T 63dc99c47a8a4f379b3349837d4b7264 P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/63dc99c47a8a4f379b3349837d4b7264
I20240503 14:13:56.637602 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet 63dc99c47a8a4f379b3349837d4b7264 (table rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]) successfully deleted
I20240503 14:13:57.925931 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:57.940250 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:57.955474 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:57.964361 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:58.003556 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:13:58.112905 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:58.115860 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:58.848088 27569 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "244c146eb7624b3585068f9c2de0c4ed"
dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50422
I20240503 14:13:58.848349 27569 raft_consensus.cc:595] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:13:58.903643 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.909677 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.917093 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.923877 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.932688 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.949900 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.958873 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.969694 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.981626 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:58.993899 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.010190 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.022953 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.040701 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.061182 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.078805 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.100311 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.122192 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.144344 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.180392 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.209095 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:13:59.235664 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:13:59.263574 30168 raft_consensus.cc:979] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: : Instructing follower aad738f0c10848789c2c811160990f8b to start an election
I20240503 14:13:59.263758 30168 raft_consensus.cc:1067] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Signalling peer aad738f0c10848789c2c811160990f8b to start an election
I20240503 14:13:59.265663 27440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "244c146eb7624b3585068f9c2de0c4ed"
dest_uuid: "aad738f0c10848789c2c811160990f8b"
from {username='slave'} at 127.20.99.3:41711
I20240503 14:13:59.265946 27440 raft_consensus.cc:482] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:59.266039 27440 raft_consensus.cc:3037] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:59.267920 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:13:59.269738 27440 raft_consensus.cc:504] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
W20240503 14:13:59.312242 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:13:59.322619 27572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244c146eb7624b3585068f9c2de0c4ed" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 581 } } ignore_live_leader: true dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:13:59.325242 27572 raft_consensus.cc:3032] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:59.325783 27572 raft_consensus.cc:729] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:13:59.326560 27572 consensus_queue.cc:260] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 581, Committed index: 581, Last appended: 1.581, Last appended by leader: 581, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:59.327235 27572 raft_consensus.cc:3037] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.329927 28735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244c146eb7624b3585068f9c2de0c4ed" candidate_uuid: "aad738f0c10848789c2c811160990f8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 581 } } ignore_live_leader: true dest_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
I20240503 14:13:59.330338 28735 raft_consensus.cc:3037] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.331043 27572 raft_consensus.cc:2454] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 2.
I20240503 14:13:59.331482 27378 leader_election.cc:302] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:13:59.331701 27440 leader_election.cc:288] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [CANDIDATE]: Term 2 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621), d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731)
I20240503 14:13:59.334601 28735 raft_consensus.cc:2454] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad738f0c10848789c2c811160990f8b in term 2.
W20240503 14:13:59.343017 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Illegal state: replica 2116de1ecc2f48daa3c8cca7edc82a95 is not leader of this config: current role FOLLOWER
I20240503 14:13:59.351434 30439 raft_consensus.cc:2781] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.351598 30439 raft_consensus.cc:686] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 2 LEADER]: Becoming Leader. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:13:59.351830 30439 consensus_queue.cc:237] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 581, Committed index: 581, Last appended: 1.581, Last appended by leader: 581, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:13:59.355957 27170 catalog_manager.cc:5484] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b reported cstate change: term changed from 1 to 2, leader changed from 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) to aad738f0c10848789c2c811160990f8b (127.20.99.2). New cstate: current_term: 2 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:13:59.789113 28733 raft_consensus.cc:1261] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 581. Preceding OpId from leader: term: 2 index: 583. (index mismatch)
I20240503 14:13:59.790341 30371 consensus_queue.cc:1035] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 582, Last known committed idx: 581, Time since last communication: 0.000s
I20240503 14:13:59.794850 27568 raft_consensus.cc:1261] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 581. Preceding OpId from leader: term: 2 index: 583. (index mismatch)
I20240503 14:13:59.795563 30371 consensus_queue.cc:1035] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 582, Last known committed idx: 581, Time since last communication: 0.000s
I20240503 14:14:00.303612 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.347224 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:00.429826 27420 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:00.440027 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.463626 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:00.473498 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:00.493366 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:01.576673 27442 consensus_queue.cc:237] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 620, Committed index: 620, Last appended: 2.620, Last appended by leader: 581, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:14:01.579600 28732 raft_consensus.cc:1261] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 620. Preceding OpId from leader: term: 2 index: 621. (index mismatch)
I20240503 14:14:01.580202 30293 consensus_queue.cc:1035] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 621, Last known committed idx: 619, Time since last communication: 0.000s
I20240503 14:14:01.584347 30574 raft_consensus.cc:2932] T 244c146eb7624b3585068f9c2de0c4ed P aad738f0c10848789c2c811160990f8b [term 2 LEADER]: Committing config change with OpId 2.621: config changed from index 91 to 621, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:01.585310 28732 raft_consensus.cc:2932] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Committing config change with OpId 2.621: config changed from index 91 to 621, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:01.596330 27170 catalog_manager.cc:5484] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 reported cstate change: config changed from index 91 to 621, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New cstate: current_term: 2 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:01.808265 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet 244c146eb7624b3585068f9c2de0c4ed with delete_type TABLET_DATA_TOMBSTONED (TS 2116de1ecc2f48daa3c8cca7edc82a95 not found in new config with opid_index 621) from {username='slave'} at 127.0.0.1:34100
I20240503 14:14:01.844166 30606 tablet_replica.cc:330] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: stopping tablet replica
I20240503 14:14:01.861913 30606 raft_consensus.cc:2229] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:01.862259 30606 raft_consensus.cc:2258] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:01.866915 30606 ts_tablet_manager.cc:1875] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:01.916379 30606 ts_tablet_manager.cc:1888] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.620
I20240503 14:14:01.916980 30606 log.cc:1196] T 244c146eb7624b3585068f9c2de0c4ed P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/244c146eb7624b3585068f9c2de0c4ed
I20240503 14:14:01.921108 27157 catalog_manager.cc:4830] TS 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): tablet 244c146eb7624b3585068f9c2de0c4ed (table rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]) successfully deleted
I20240503 14:14:02.824283 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:02.858544 27291 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:03.035144 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.096017 27420 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:03.153605 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.219820 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:03.223412 28584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:03.615469 27308 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f3923478811a4a0f82ec9f013ef7735d"
dest_uuid: "60da896b8369489994355cc0e9050514"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51002
I20240503 14:14:03.615736 27308 raft_consensus.cc:595] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:03.678046 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.686865 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.712214 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.729151 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.757330 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.778344 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.792944 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.804782 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.816694 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.838924 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.852293 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.866585 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.885133 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.902695 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.925906 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.946184 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.967875 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
W20240503 14:14:03.994907 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:14:04.025166 30113 raft_consensus.cc:979] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: : Instructing follower c57eaae6a75a43679120a10c6d3090c7 to start an election
I20240503 14:14:04.025334 30681 raft_consensus.cc:1067] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Signalling peer c57eaae6a75a43679120a10c6d3090c7 to start an election
I20240503 14:14:04.027336 28603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3923478811a4a0f82ec9f013ef7735d"
dest_uuid: "c57eaae6a75a43679120a10c6d3090c7"
from {username='slave'} at 127.20.99.1:36175
I20240503 14:14:04.027679 28603 raft_consensus.cc:482] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:04.027954 28603 raft_consensus.cc:3037] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:04.030702 28603 raft_consensus.cc:504] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
W20240503 14:14:04.034905 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Service unavailable: leader transfer in progress
I20240503 14:14:04.040385 28603 leader_election.cc:288] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [CANDIDATE]: Term 2 election: Requested vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565), aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:14:04.040941 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3923478811a4a0f82ec9f013ef7735d" candidate_uuid: "c57eaae6a75a43679120a10c6d3090c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 655 } } ignore_live_leader: true dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:14:04.041257 27310 raft_consensus.cc:3032] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:04.041407 27310 raft_consensus.cc:729] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:14:04.041762 27310 consensus_queue.cc:260] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 655, Committed index: 655, Last appended: 1.655, Last appended by leader: 655, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:04.042205 27310 raft_consensus.cc:3037] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:04.044978 27310 raft_consensus.cc:2454] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57eaae6a75a43679120a10c6d3090c7 in term 2.
I20240503 14:14:04.047024 28537 leader_election.cc:302] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, c57eaae6a75a43679120a10c6d3090c7; no voters:
W20240503 14:14:04.057994 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Illegal state: replica 60da896b8369489994355cc0e9050514 is not leader of this config: current role FOLLOWER
I20240503 14:14:04.063331 30704 raft_consensus.cc:2781] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:14:04.088416 27399 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46712: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
W20240503 14:14:04.121172 27269 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57930: Illegal state: replica 60da896b8369489994355cc0e9050514 is not leader of this config: current role FOLLOWER
I20240503 14:14:04.148371 30704 raft_consensus.cc:686] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 2 LEADER]: Becoming Leader. State: Replica: c57eaae6a75a43679120a10c6d3090c7, State: Running, Role: LEADER
I20240503 14:14:04.149155 30704 consensus_queue.cc:237] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 655, Committed index: 655, Last appended: 1.655, Last appended by leader: 655, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:04.152478 27442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3923478811a4a0f82ec9f013ef7735d" candidate_uuid: "c57eaae6a75a43679120a10c6d3090c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 655 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:14:04.152670 27442 raft_consensus.cc:3037] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:04.156745 27161 catalog_manager.cc:5484] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 reported cstate change: term changed from 1 to 2, leader changed from 60da896b8369489994355cc0e9050514 (127.20.99.1) to c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4). New cstate: current_term: 2 leader_uuid: "c57eaae6a75a43679120a10c6d3090c7" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
W20240503 14:14:04.157608 27399 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46712: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:14:04.157892 27442 raft_consensus.cc:2454] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57eaae6a75a43679120a10c6d3090c7 in term 2.
W20240503 14:14:04.260216 27507 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 48602.2us
W20240503 14:14:04.374112 28929 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 76352.4us
I20240503 14:14:04.382579 27308 raft_consensus.cc:1261] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Refusing update from remote peer c57eaae6a75a43679120a10c6d3090c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 655. Preceding OpId from leader: term: 2 index: 657. (index mismatch)
I20240503 14:14:04.383639 30708 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 656, Last known committed idx: 655, Time since last communication: 0.000s
I20240503 14:14:04.391561 30613 mvcc.cc:205] Tried to move back new op lower bound from 7023598159378468864 to 7023598158441488384. Current Snapshot: MvccSnapshot[applied={T|T < 7023598159378468864 or (T in {7023598159378468864})}]
I20240503 14:14:04.395540 27442 raft_consensus.cc:1261] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer c57eaae6a75a43679120a10c6d3090c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 655. Preceding OpId from leader: term: 2 index: 657. (index mismatch)
I20240503 14:14:04.405208 30704 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 656, Last known committed idx: 655, Time since last communication: 0.000s
I20240503 14:14:04.451004 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:04.494907 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:04.509281 27420 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:04.523617 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:04.577234 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:04.578699 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:04.578960 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:05.210906 27507 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 73101.9us
I20240503 14:14:05.549309 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:05.618975 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:05.619915 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:05.642897 27420 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:14:05.644487 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:05.685047 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:05.708364 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:14:05.915299 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:37731, user_credentials={real_user=slave}} blocked reactor thread for 44634.6us
I20240503 14:14:06.334690 27440 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd"
dest_uuid: "aad738f0c10848789c2c811160990f8b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50734
I20240503 14:14:06.335062 27440 raft_consensus.cc:595] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:06.396721 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.402662 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.406378 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.413801 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.425491 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.437266 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.450582 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.459481 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.475324 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.490303 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.507164 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.523078 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.542037 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.556721 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.584400 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.604123 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.627367 27399 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.651582 27399 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.674690 27399 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:06.700088 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:06.713888 30574 raft_consensus.cc:979] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: : Instructing follower 2116de1ecc2f48daa3c8cca7edc82a95 to start an election
I20240503 14:14:06.714047 30574 raft_consensus.cc:1067] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Signalling peer 2116de1ecc2f48daa3c8cca7edc82a95 to start an election
I20240503 14:14:06.714942 27569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd"
dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
from {username='slave'} at 127.20.99.2:33741
I20240503 14:14:06.715159 27569 raft_consensus.cc:482] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:06.715270 27569 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:06.718073 27569 raft_consensus.cc:504] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:06.719266 27440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 2 candidate_status { last_received { term: 1 index: 565 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:14:06.719477 27440 raft_consensus.cc:3032] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:06.719564 27440 raft_consensus.cc:729] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:14:06.719890 27440 consensus_queue.cc:260] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 565, Committed index: 565, Last appended: 1.565, Last appended by leader: 565, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:06.721329 27440 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:14:06.724295 27402 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:14:06.727360 28995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" candidate_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" candidate_term: 2 candidate_status { last_received { term: 1 index: 565 } } ignore_live_leader: true dest_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
I20240503 14:14:06.727689 28995 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:06.728145 27440 raft_consensus.cc:2454] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 2.
I20240503 14:14:06.730465 28995 raft_consensus.cc:2454] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2116de1ecc2f48daa3c8cca7edc82a95 in term 2.
I20240503 14:14:06.731580 27506 leader_election.cc:302] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2116de1ecc2f48daa3c8cca7edc82a95, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:14:06.731968 30750 raft_consensus.cc:2781] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:06.732095 30750 raft_consensus.cc:686] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Becoming Leader. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:14:06.732352 30750 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 565, Committed index: 565, Last appended: 1.565, Last appended by leader: 565, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:06.735422 27165 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: term changed from 1 to 2, leader changed from aad738f0c10848789c2c811160990f8b (127.20.99.2) to 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3). New cstate: current_term: 2 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:06.744858 27569 leader_election.cc:288] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [CANDIDATE]: Term 2 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553), dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199)
W20240503 14:14:06.907984 29060 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.20.99.3:43621, user_credentials={real_user=slave}} blocked reactor thread for 65024.2us
I20240503 14:14:07.082427 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:07.094957 28995 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 565. Preceding OpId from leader: term: 2 index: 567. (index mismatch)
I20240503 14:14:07.096145 30518 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 566, Last known committed idx: 565, Time since last communication: 0.000s
W20240503 14:14:07.112102 28537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 59391.7us
I20240503 14:14:07.146875 27442 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 1 index: 565. Preceding OpId from leader: term: 2 index: 567. (index mismatch)
I20240503 14:14:07.148505 30875 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 566, Last known committed idx: 565, Time since last communication: 0.000s
I20240503 14:14:07.150470 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:07.260792 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:07.265710 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:07.316484 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:07.395495 27376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:45729, user_credentials={real_user=slave}} blocked reactor thread for 105177us
I20240503 14:14:07.400588 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:07.438601 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:08.030059 27568 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 578, Committed index: 578, Last appended: 2.579, Last appended by leader: 565, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:08.034910 28993 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 579. Preceding OpId from leader: term: 2 index: 580. (index mismatch)
I20240503 14:14:08.037221 30853 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 580, Last known committed idx: 578, Time since last communication: 0.000s
I20240503 14:14:08.041489 30853 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Committing config change with OpId 2.580: config changed from index 81 to 580, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:08.043291 28995 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Committing config change with OpId 2.580: config changed from index 81 to 580, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:08.052493 27165 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 81 to 580, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
W20240503 14:14:08.249997 27376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:45729, user_credentials={real_user=slave}} blocked reactor thread for 60656.7us
I20240503 14:14:08.345640 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet e8102c1d08074ea7bb5be6c5645d1fcd with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 580) from {username='slave'} at 127.0.0.1:50958
I20240503 14:14:08.381721 30933 tablet_replica.cc:330] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:14:08.383847 30933 raft_consensus.cc:2229] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:08.384443 30933 pending_rounds.cc:62] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Trying to abort 1 pending ops.
I20240503 14:14:08.384871 30933 pending_rounds.cc:68] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Aborting op as it isn't in flight: id { term: 2 index: 579 } timestamp: 7023598174309863424 op_type: WRITE_OP write_request { tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000Z&\355#\214D\347\300\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4a6864739735c05e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023598174220898304 authz_token { token_data: "\010\264\344\323\261\006\"3\n\005slave\022*\n 5ef2a6e4b3704917a5e6e7baf23cb90b\020\001\030\001 \001(\001" signature: "<redacted>""\351c,\330D\325\307}\021\t\256\371^at\332\205\350vB\3445\001\325\312\255Jh\256N\037\0362\000\007\003W\255\350\220\252\031\234\261\245\315gQ\020\357=\340\270\nr<\271\212\366f\211\013\341G" signing_key_seq_num: 0 } } request_id { client_id: "f51072c25a2242ab8ce8a1c727024758" seq_no: 1678 first_incomplete_seq_no: 1678 attempt_no: 0 }
I20240503 14:14:08.386922 30933 raft_consensus.cc:2258] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:08.390779 30933 ts_tablet_manager.cc:1875] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:08.427521 30933 ts_tablet_manager.cc:1888] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.579
I20240503 14:14:08.430851 30933 log.cc:1196] T e8102c1d08074ea7bb5be6c5645d1fcd P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:14:08.448835 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet e8102c1d08074ea7bb5be6c5645d1fcd (table rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]) successfully deleted
I20240503 14:14:09.345535 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:09.386366 27552 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:09.483301 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:09.576283 28975 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:09.583524 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:09.602563 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:09.691115 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:09.821767 27507 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 59886.1us
I20240503 14:14:10.141340 27569 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a352cdeffa9a4b85badcefa545885bc3"
dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33392
I20240503 14:14:10.141558 27569 raft_consensus.cc:595] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:10.296072 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:14:10.304450 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:14:10.309319 30984 raft_consensus.cc:979] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: : Instructing follower dac9fd6f6a404989ae1a460ac3cde0fe to start an election
I20240503 14:14:10.309468 30984 raft_consensus.cc:1067] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Signalling peer dac9fd6f6a404989ae1a460ac3cde0fe to start an election
W20240503 14:14:10.315549 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
W20240503 14:14:10.328612 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:14:10.332170 28993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a352cdeffa9a4b85badcefa545885bc3"
dest_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
from {username='slave'} at 127.20.99.3:34287
I20240503 14:14:10.332490 28993 raft_consensus.cc:482] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:10.333621 28993 raft_consensus.cc:3037] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:14:10.336733 27530 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:14:10.347955 28993 raft_consensus.cc:504] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
W20240503 14:14:10.350875 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Service unavailable: leader transfer in progress
I20240503 14:14:10.353046 28993 leader_election.cc:288] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621), 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
I20240503 14:14:10.356469 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a352cdeffa9a4b85badcefa545885bc3" candidate_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 763 } } ignore_live_leader: true dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:14:10.356657 27310 raft_consensus.cc:3037] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:10.359275 27310 raft_consensus.cc:2454] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac9fd6f6a404989ae1a460ac3cde0fe in term 2.
I20240503 14:14:10.359755 28928 leader_election.cc:302] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60da896b8369489994355cc0e9050514, dac9fd6f6a404989ae1a460ac3cde0fe; no voters:
I20240503 14:14:10.360070 30724 raft_consensus.cc:2781] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:10.360203 30724 raft_consensus.cc:686] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 LEADER]: Becoming Leader. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Running, Role: LEADER
I20240503 14:14:10.360520 30724 consensus_queue.cc:237] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 763, Committed index: 763, Last appended: 1.763, Last appended by leader: 763, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:10.360782 27568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a352cdeffa9a4b85badcefa545885bc3" candidate_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 763 } } ignore_live_leader: true dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:14:10.360980 27568 raft_consensus.cc:3032] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:10.361056 27568 raft_consensus.cc:729] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:14:10.361456 27568 consensus_queue.cc:260] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 763, Committed index: 763, Last appended: 1.763, Last appended by leader: 763, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:10.361835 27568 raft_consensus.cc:3037] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:10.363010 27170 catalog_manager.cc:5484] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: term changed from 1 to 2, leader changed from 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) to dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7). New cstate: current_term: 2 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 157 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:10.365278 27568 raft_consensus.cc:2454] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac9fd6f6a404989ae1a460ac3cde0fe in term 2.
W20240503 14:14:10.368276 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40878: Illegal state: replica 2116de1ecc2f48daa3c8cca7edc82a95 is not leader of this config: current role FOLLOWER
W20240503 14:14:10.510845 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43726: Illegal state: replica 60da896b8369489994355cc0e9050514 is not leader of this config: current role FOLLOWER
I20240503 14:14:10.557183 27568 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 763. Preceding OpId from leader: term: 2 index: 765. (index mismatch)
I20240503 14:14:10.557922 30720 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 764, Last known committed idx: 763, Time since last communication: 0.000s
I20240503 14:14:10.558107 27311 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 763. Preceding OpId from leader: term: 2 index: 765. (index mismatch)
I20240503 14:14:10.559228 30720 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 764, Last known committed idx: 763, Time since last communication: 0.000s
W20240503 14:14:10.659560 28931 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:43621, user_credentials={real_user=slave}} blocked reactor thread for 60603.4us
I20240503 14:14:11.067690 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:11.089217 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:11.128895 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:11.170361 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:11.199651 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:11.521850 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:11.537998 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:14:12.008423 28929 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 67559.1us
W20240503 14:14:12.290369 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:37731, user_credentials={real_user=slave}} blocked reactor thread for 53067us
I20240503 14:14:12.428833 28993 consensus_queue.cc:237] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 794, Committed index: 794, Last appended: 2.794, Last appended by leader: 763, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 795 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:14:12.432662 27307 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 794. Preceding OpId from leader: term: 2 index: 795. (index mismatch)
I20240503 14:14:12.435002 31077 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 795, Last known committed idx: 794, Time since last communication: 0.000s
I20240503 14:14:12.439603 31077 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 LEADER]: Committing config change with OpId 2.795: config changed from index 157 to 795, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 795 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:12.443696 27307 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Committing config change with OpId 2.795: config changed from index 157 to 795, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 795 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:12.447978 27165 catalog_manager.cc:5484] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 157 to 795, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New cstate: current_term: 2 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 795 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } }
I20240503 14:14:12.515877 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet a352cdeffa9a4b85badcefa545885bc3 with delete_type TABLET_DATA_TOMBSTONED (TS 2116de1ecc2f48daa3c8cca7edc82a95 not found in new config with opid_index 795) from {username='slave'} at 127.0.0.1:34100
I20240503 14:14:12.540593 31111 tablet_replica.cc:330] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: stopping tablet replica
I20240503 14:14:12.541182 31111 raft_consensus.cc:2229] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:12.541488 31111 raft_consensus.cc:2258] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:12.560525 31111 ts_tablet_manager.cc:1875] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:12.581967 31111 ts_tablet_manager.cc:1888] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.794
I20240503 14:14:12.582237 31111 log.cc:1196] T a352cdeffa9a4b85badcefa545885bc3 P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/a352cdeffa9a4b85badcefa545885bc3
I20240503 14:14:12.583719 27157 catalog_manager.cc:4830] TS 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): tablet a352cdeffa9a4b85badcefa545885bc3 (table rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]) successfully deleted
I20240503 14:14:13.526161 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:13.631584 28845 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:13.641388 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:13.655107 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:13.755236 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:13.788622 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:13.872819 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:14.162077 27310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f576c9fe2ddf4494b51d41d64d18a1eb"
dest_uuid: "60da896b8369489994355cc0e9050514"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43762
I20240503 14:14:14.162294 27310 raft_consensus.cc:595] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:14.279585 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.286636 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.294275 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.299696 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.309607 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.321309 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.340893 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.354717 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.366182 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.380692 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
W20240503 14:14:14.397156 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
I20240503 14:14:14.402839 31070 raft_consensus.cc:979] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: : Instructing follower 01c7323dc6f84fe0bc5eca4fcad8f078 to start an election
I20240503 14:14:14.402961 31070 raft_consensus.cc:1067] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Signalling peer 01c7323dc6f84fe0bc5eca4fcad8f078 to start an election
I20240503 14:14:14.404381 28863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f576c9fe2ddf4494b51d41d64d18a1eb"
dest_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
from {username='slave'} at 127.20.99.1:48133
I20240503 14:14:14.404698 28863 raft_consensus.cc:482] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:14.404906 28863 raft_consensus.cc:3037] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:14.407783 28863 raft_consensus.cc:504] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:14.411643 28863 leader_election.cc:288] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 2 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621), 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
W20240503 14:14:14.413547 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Service unavailable: leader transfer in progress
I20240503 14:14:14.413861 27310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f576c9fe2ddf4494b51d41d64d18a1eb" candidate_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" candidate_term: 2 candidate_status { last_received { term: 1 index: 716 } } ignore_live_leader: true dest_uuid: "60da896b8369489994355cc0e9050514"
I20240503 14:14:14.414045 27310 raft_consensus.cc:3032] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:14.414132 27310 raft_consensus.cc:729] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:14:14.414386 27310 consensus_queue.cc:260] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 716, Committed index: 716, Last appended: 1.716, Last appended by leader: 716, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:14.414722 27310 raft_consensus.cc:3037] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:14.418081 27310 raft_consensus.cc:2454] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c7323dc6f84fe0bc5eca4fcad8f078 in term 2.
I20240503 14:14:14.418696 28798 leader_election.cc:302] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c7323dc6f84fe0bc5eca4fcad8f078, 60da896b8369489994355cc0e9050514; no voters:
W20240503 14:14:14.429589 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Illegal state: replica 60da896b8369489994355cc0e9050514 is not leader of this config: current role FOLLOWER
I20240503 14:14:14.452544 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f576c9fe2ddf4494b51d41d64d18a1eb" candidate_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" candidate_term: 2 candidate_status { last_received { term: 1 index: 716 } } ignore_live_leader: true dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:14:14.452800 27571 raft_consensus.cc:3037] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:14.452942 31201 raft_consensus.cc:2781] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:14.455520 27571 raft_consensus.cc:2454] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c7323dc6f84fe0bc5eca4fcad8f078 in term 2.
I20240503 14:14:14.469874 31201 raft_consensus.cc:686] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 LEADER]: Becoming Leader. State: Replica: 01c7323dc6f84fe0bc5eca4fcad8f078, State: Running, Role: LEADER
I20240503 14:14:14.470551 31201 consensus_queue.cc:237] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 716, Committed index: 716, Last appended: 1.716, Last appended by leader: 716, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:14.474841 27170 catalog_manager.cc:5484] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 reported cstate change: term changed from 1 to 2, leader changed from 60da896b8369489994355cc0e9050514 (127.20.99.1) to 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6). New cstate: current_term: 2 leader_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
W20240503 14:14:14.477399 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33426: Illegal state: replica 2116de1ecc2f48daa3c8cca7edc82a95 is not leader of this config: current role FOLLOWER
W20240503 14:14:14.558575 27266 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57934: Illegal state: replica 60da896b8369489994355cc0e9050514 is not leader of this config: current role FOLLOWER
W20240503 14:14:14.576208 27531 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33426: Illegal state: replica 2116de1ecc2f48daa3c8cca7edc82a95 is not leader of this config: current role FOLLOWER
I20240503 14:14:14.594071 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:14.612576 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:14.623802 27311 raft_consensus.cc:1261] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 1 index: 716. Preceding OpId from leader: term: 2 index: 718. (index mismatch)
I20240503 14:14:14.625320 31207 consensus_queue.cc:1035] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 717, Last known committed idx: 716, Time since last communication: 0.000s
I20240503 14:14:14.635387 27568 raft_consensus.cc:1261] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 1 index: 716. Preceding OpId from leader: term: 2 index: 718. (index mismatch)
I20240503 14:14:14.636806 31207 consensus_queue.cc:1035] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 717, Last known committed idx: 716, Time since last communication: 0.000s
I20240503 14:14:14.637706 31138 mvcc.cc:205] Tried to move back new op lower bound from 7023598201318809600 to 7023598200718086144. Current Snapshot: MvccSnapshot[applied={T|T < 7023598201318809600}]
I20240503 14:14:14.714154 27291 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:14.828405 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:14.883927 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:14.892701 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:15.232975 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:16.328401 27291 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:16.362253 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:16.375645 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:16.445179 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:16.464852 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:16.498732 27420 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:14:16.521255 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:17.166220 27442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4180d30f1d3449ff9c28c69599098ff4"
dest_uuid: "aad738f0c10848789c2c811160990f8b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49950
I20240503 14:14:17.167080 27442 raft_consensus.cc:595] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:17.224388 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.228590 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.239495 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.247753 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.259805 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.268644 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.276649 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.288379 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.302968 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.314144 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.329957 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.342584 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.363075 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.379007 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.397609 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.419085 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.441152 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.464351 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.488961 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.514099 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.552367 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.579059 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.605723 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.632403 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.661141 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:17.691576 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:17.705989 31074 raft_consensus.cc:979] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: : Instructing follower 60da896b8369489994355cc0e9050514 to start an election
I20240503 14:14:17.706318 31074 raft_consensus.cc:1067] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Signalling peer 60da896b8369489994355cc0e9050514 to start an election
I20240503 14:14:17.707351 27307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4180d30f1d3449ff9c28c69599098ff4"
dest_uuid: "60da896b8369489994355cc0e9050514"
from {username='slave'} at 127.20.99.2:37943
I20240503 14:14:17.707639 27307 raft_consensus.cc:482] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:17.707806 27307 raft_consensus.cc:3037] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:14:17.721335 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:17.742980 27307 raft_consensus.cc:504] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
W20240503 14:14:17.754500 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:17.763932 27307 leader_election.cc:288] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 2 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553), 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729)
I20240503 14:14:17.765740 28863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4180d30f1d3449ff9c28c69599098ff4" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 2 candidate_status { last_received { term: 1 index: 681 } } ignore_live_leader: true dest_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
I20240503 14:14:17.766096 28863 raft_consensus.cc:3037] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:17.769009 28863 raft_consensus.cc:2454] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 2.
I20240503 14:14:17.772629 27245 leader_election.cc:302] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c7323dc6f84fe0bc5eca4fcad8f078, 60da896b8369489994355cc0e9050514; no voters:
W20240503 14:14:17.772869 28798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 72605.9us
I20240503 14:14:17.775252 31119 raft_consensus.cc:2781] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:17.775420 31119 raft_consensus.cc:686] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 2 LEADER]: Becoming Leader. State: Replica: 60da896b8369489994355cc0e9050514, State: Running, Role: LEADER
I20240503 14:14:17.775696 31119 consensus_queue.cc:237] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 681, Committed index: 681, Last appended: 1.681, Last appended by leader: 681, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:17.782137 27170 catalog_manager.cc:5484] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 reported cstate change: term changed from 1 to 2, leader changed from aad738f0c10848789c2c811160990f8b (127.20.99.2) to 60da896b8369489994355cc0e9050514 (127.20.99.1). New cstate: current_term: 2 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
W20240503 14:14:17.784672 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:17.800040 27440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4180d30f1d3449ff9c28c69599098ff4" candidate_uuid: "60da896b8369489994355cc0e9050514" candidate_term: 2 candidate_status { last_received { term: 1 index: 681 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:14:17.800288 27440 raft_consensus.cc:3032] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:17.800390 27440 raft_consensus.cc:729] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:14:17.803555 27440 consensus_queue.cc:260] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 681, Committed index: 681, Last appended: 1.681, Last appended by leader: 681, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:17.803946 27440 raft_consensus.cc:3037] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:17.808815 27440 raft_consensus.cc:2454] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60da896b8369489994355cc0e9050514 in term 2.
W20240503 14:14:17.815950 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:14:17.826277 28863 raft_consensus.cc:1226] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Rejecting Update request from peer aad738f0c10848789c2c811160990f8b for earlier term 1. Current term is 2. Ops: []
I20240503 14:14:17.950409 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:17.964648 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:17.974136 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:17.996060 28863 raft_consensus.cc:1261] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 681. Preceding OpId from leader: term: 2 index: 683. (index mismatch)
I20240503 14:14:17.996850 30680 consensus_queue.cc:1035] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 682, Last known committed idx: 681, Time since last communication: 0.000s
I20240503 14:14:18.005396 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:18.009660 27440 raft_consensus.cc:1261] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 1 index: 681. Preceding OpId from leader: term: 2 index: 683. (index mismatch)
I20240503 14:14:18.019645 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:18.033882 31311 consensus_queue.cc:1035] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 682, Last known committed idx: 681, Time since last communication: 0.000s
I20240503 14:14:18.064093 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:18.213145 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:18.543336 27378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:43621, user_credentials={real_user=slave}} blocked reactor thread for 57864us
I20240503 14:14:19.280258 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:19.303928 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:19.360888 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:19.400029 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:19.419241 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:19.450381 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:19.470897 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:20.153944 27442 consensus_queue.cc:237] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 831, Committed index: 831, Last appended: 1.831, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 832 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:14:20.155762 28733 raft_consensus.cc:1261] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 831. Preceding OpId from leader: term: 1 index: 832. (index mismatch)
I20240503 14:14:20.156584 30880 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 832, Last known committed idx: 831, Time since last communication: 0.000s
I20240503 14:14:20.167932 30883 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.832: config changed from index 105 to 832, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 832 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:20.172365 28732 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Committing config change with OpId 1.832: config changed from index 105 to 832, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 832 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:20.178947 27167 catalog_manager.cc:5484] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 105 to 832, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 832 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:20.267508 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet 98bec1722172403aa21b3d060ed636d2 with delete_type TABLET_DATA_TOMBSTONED (TS 2116de1ecc2f48daa3c8cca7edc82a95 not found in new config with opid_index 832) from {username='slave'} at 127.0.0.1:34100
I20240503 14:14:20.287509 31493 tablet_replica.cc:330] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: stopping tablet replica
I20240503 14:14:20.302325 31493 raft_consensus.cc:2229] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:20.302613 31493 raft_consensus.cc:2258] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:20.307897 31493 ts_tablet_manager.cc:1875] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:20.338833 31493 ts_tablet_manager.cc:1888] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.831
I20240503 14:14:20.339321 31493 log.cc:1196] T 98bec1722172403aa21b3d060ed636d2 P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/98bec1722172403aa21b3d060ed636d2
I20240503 14:14:20.340530 27157 catalog_manager.cc:4830] TS 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): tablet 98bec1722172403aa21b3d060ed636d2 (table rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]) successfully deleted
I20240503 14:14:22.280448 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:22.319800 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:22.357818 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:22.438266 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:22.474490 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:22.478719 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:22.566269 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:23.097328 28601 consensus_queue.cc:237] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1070, Committed index: 1070, Last appended: 2.1070, Last appended by leader: 655, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1071 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:23.106392 27442 raft_consensus.cc:1261] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer c57eaae6a75a43679120a10c6d3090c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 1070. Preceding OpId from leader: term: 2 index: 1071. (index mismatch)
I20240503 14:14:23.107197 31330 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1071, Last known committed idx: 1070, Time since last communication: 0.000s
I20240503 14:14:23.110303 31330 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 2 LEADER]: Committing config change with OpId 2.1071: config changed from index 108 to 1071, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 1071 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:23.113142 27442 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Committing config change with OpId 2.1071: config changed from index 108 to 1071, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 1071 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:23.142074 27167 catalog_manager.cc:5484] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 reported cstate change: config changed from index 108 to 1071, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "c57eaae6a75a43679120a10c6d3090c7" committed_config { opid_index: 1071 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:23.247884 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet f3923478811a4a0f82ec9f013ef7735d with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 1071) from {username='slave'} at 127.0.0.1:60240
I20240503 14:14:23.293123 31647 tablet_replica.cc:330] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:14:23.293334 31647 raft_consensus.cc:2229] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:23.293535 31647 raft_consensus.cc:2258] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:23.381945 31647 ts_tablet_manager.cc:1875] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:23.441979 31647 ts_tablet_manager.cc:1888] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1070
I20240503 14:14:23.444347 31647 log.cc:1196] T f3923478811a4a0f82ec9f013ef7735d P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/f3923478811a4a0f82ec9f013ef7735d
I20240503 14:14:23.451093 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet f3923478811a4a0f82ec9f013ef7735d (table rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]) successfully deleted
I20240503 14:14:24.662089 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:24.700661 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:24.705585 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:24.799269 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:24.811537 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:24.827458 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:24.840863 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:25.756227 28865 consensus_queue.cc:237] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 936, Committed index: 936, Last appended: 2.936, Last appended by leader: 716, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 937 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:25.762900 27572 raft_consensus.cc:1261] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 2 index: 936. Preceding OpId from leader: term: 2 index: 937. (index mismatch)
I20240503 14:14:25.763628 31694 consensus_queue.cc:1035] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 937, Last known committed idx: 936, Time since last communication: 0.000s
I20240503 14:14:25.766336 31723 raft_consensus.cc:2932] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 LEADER]: Committing config change with OpId 2.937: config changed from index 104 to 937, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 937 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:25.766878 27572 raft_consensus.cc:2932] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Committing config change with OpId 2.937: config changed from index 104 to 937, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 937 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:25.775928 27167 catalog_manager.cc:5484] T f576c9fe2ddf4494b51d41d64d18a1eb P 2116de1ecc2f48daa3c8cca7edc82a95 reported cstate change: config changed from index 104 to 937, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" committed_config { opid_index: 937 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:25.838485 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet f576c9fe2ddf4494b51d41d64d18a1eb with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 937) from {username='slave'} at 127.0.0.1:60240
I20240503 14:14:25.847553 31761 tablet_replica.cc:330] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:14:25.847761 31761 raft_consensus.cc:2229] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:25.847966 31761 raft_consensus.cc:2258] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:25.849504 31761 ts_tablet_manager.cc:1875] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:25.868316 31761 ts_tablet_manager.cc:1888] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.936
I20240503 14:14:25.878528 31761 log.cc:1196] T f576c9fe2ddf4494b51d41d64d18a1eb P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/f576c9fe2ddf4494b51d41d64d18a1eb
I20240503 14:14:25.880816 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet f576c9fe2ddf4494b51d41d64d18a1eb (table rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]) successfully deleted
I20240503 14:14:26.535663 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:26.549937 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:26.617245 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:26.644951 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:26.660404 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:26.732769 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:26.762768 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:27.242764 27307 consensus_queue.cc:237] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 837, Committed index: 837, Last appended: 2.838, Last appended by leader: 681, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:27.245824 28864 raft_consensus.cc:1261] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Refusing update from remote peer 60da896b8369489994355cc0e9050514: Log matching property violated. Preceding OpId in replica: term: 2 index: 838. Preceding OpId from leader: term: 2 index: 839. (index mismatch)
I20240503 14:14:27.246577 31514 consensus_queue.cc:1035] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 839, Last known committed idx: 838, Time since last communication: 0.000s
I20240503 14:14:27.250016 31515 raft_consensus.cc:2932] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 [term 2 LEADER]: Committing config change with OpId 2.839: config changed from index 61 to 839, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:27.252971 28864 raft_consensus.cc:2932] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Committing config change with OpId 2.839: config changed from index 61 to 839, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:27.256378 27167 catalog_manager.cc:5484] T 4180d30f1d3449ff9c28c69599098ff4 P 60da896b8369489994355cc0e9050514 reported cstate change: config changed from index 61 to 839, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "60da896b8369489994355cc0e9050514" committed_config { opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:27.299437 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet 4180d30f1d3449ff9c28c69599098ff4 with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 839) from {username='slave'} at 127.0.0.1:50958
I20240503 14:14:27.332381 31839 tablet_replica.cc:330] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:14:27.332588 31839 raft_consensus.cc:2229] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:27.332792 31839 pending_rounds.cc:62] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Trying to abort 1 pending ops.
I20240503 14:14:27.332861 31839 pending_rounds.cc:68] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Aborting op as it isn't in flight: id { term: 2 index: 838 } timestamp: 7023598252987645952 op_type: WRITE_OP write_request { tablet_id: "4180d30f1d3449ff9c28c69599098ff4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\204\004\342\034\000\300\360o\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""fdef190638a7f15c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023598252902375424 authz_token { token_data: "\010\264\344\323\261\006\"3\n\005slave\022*\n 5ef2a6e4b3704917a5e6e7baf23cb90b\020\001\030\001 \001(\001" signature: "<redacted>""\351c,\330D\325\307}\021\t\256\371^at\332\205\350vB\3445\001\325\312\255Jh\256N\037\0362\000\007\003W\255\350\220\252\031\234\261\245\315gQ\020\357=\340\270\nr<\271\212\366f\211\013\341G" signing_key_seq_num: 0 } } request_id { client_id: "f51072c25a2242ab8ce8a1c727024758" seq_no: 2621 first_incomplete_seq_no: 2621 attempt_no: 0 }
I20240503 14:14:27.333353 31839 raft_consensus.cc:2258] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:27.343806 31839 ts_tablet_manager.cc:1875] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:27.365450 31839 ts_tablet_manager.cc:1888] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.838
I20240503 14:14:27.365695 31839 log.cc:1196] T 4180d30f1d3449ff9c28c69599098ff4 P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/4180d30f1d3449ff9c28c69599098ff4
I20240503 14:14:27.368638 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet 4180d30f1d3449ff9c28c69599098ff4 (table rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]) successfully deleted
I20240503 14:14:27.603436 27440 consensus_queue.cc:237] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 912, Committed index: 912, Last appended: 1.913, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 914 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:14:27.606418 27307 raft_consensus.cc:1261] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 913. Preceding OpId from leader: term: 1 index: 914. (index mismatch)
I20240503 14:14:27.607144 31724 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 914, Last known committed idx: 912, Time since last communication: 0.000s
W20240503 14:14:27.609436 27375 consensus_peers.cc:485] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b -> Peer d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731): Couldn't send request to peer d7a1a0df5dd047ddbacafd0870ed01d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c65982ba3034daeacd3407a5b6877e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:27.612726 31724 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.914: config changed from index -1 to 914, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 914 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:27.613323 27307 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Committing config change with OpId 1.914: config changed from index -1 to 914, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New config: { opid_index: 914 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:27.639694 27170 catalog_manager.cc:5484] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index -1 to 914, VOTER d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 914 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
W20240503 14:14:27.702847 28928 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 54626.5us
I20240503 14:14:28.136693 31883 ts_tablet_manager.cc:902] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:14:28.195547 31883 tablet_copy_client.cc:308] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:14:28.197875 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet 2c65982ba3034daeacd3407a5b6877e1 from peer d7a1a0df5dd047ddbacafd0870ed01d1 ({username='slave'} at 127.20.99.5:52387)
I20240503 14:14:28.198077 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet 2c65982ba3034daeacd3407a5b6877e1 from peer d7a1a0df5dd047ddbacafd0870ed01d1 at {username='slave'} at 127.20.99.5:52387: session id = d7a1a0df5dd047ddbacafd0870ed01d1-2c65982ba3034daeacd3407a5b6877e1
I20240503 14:14:28.201244 27462 tablet_copy_source_session.cc:205] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:28.204167 31883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c65982ba3034daeacd3407a5b6877e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:28.228495 31883 tablet_copy_client.cc:783] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:28.232152 31883 tablet_copy_client.cc:647] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:28.254525 31883 tablet_copy_client.cc:523] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:28.260684 31883 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap starting.
I20240503 14:14:28.824007 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:28.884286 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:28.926074 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:28.991902 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:29.018527 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:29.059752 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:29.177507 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:30.502071 31883 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=926 overwritten=0 applied=926 ignored=0} inserts{seen=1611 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:30.504657 31883 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: Bootstrap complete.
I20240503 14:14:30.505299 31883 ts_tablet_manager.cc:1367] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent bootstrapping tablet: real 2.245s user 1.089s sys 0.105s
I20240503 14:14:30.506529 31883 raft_consensus.cc:348] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 914 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:14:30.506942 31883 raft_consensus.cc:729] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a1a0df5dd047ddbacafd0870ed01d1, State: Initialized, Role: FOLLOWER
W20240503 14:14:30.507426 27375 consensus_peers.cc:485] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b -> Peer d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731): Couldn't send request to peer d7a1a0df5dd047ddbacafd0870ed01d1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:30.507345 31883 consensus_queue.cc:260] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 926, Last appended: 1.926, Last appended by leader: 926, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 914 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:14:30.521924 31883 ts_tablet_manager.cc:1398] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20240503 14:14:30.522934 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-2c65982ba3034daeacd3407a5b6877e1 received from {username='slave'} at 127.20.99.5:52387
I20240503 14:14:30.523101 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session d7a1a0df5dd047ddbacafd0870ed01d1-2c65982ba3034daeacd3407a5b6877e1 on tablet 2c65982ba3034daeacd3407a5b6877e1 with peer d7a1a0df5dd047ddbacafd0870ed01d1
I20240503 14:14:30.971561 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:30.988411 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:31.074357 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:31.079588 28734 raft_consensus.cc:1203] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.913->[1.914-1.977] Dedup: 1.926->[1.927-1.977]
I20240503 14:14:31.085731 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:31.093638 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:31.284044 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:31.459288 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:14:31.792588 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:45565, user_credentials={real_user=slave}} blocked reactor thread for 43883.3us
I20240503 14:14:31.930720 27440 consensus_queue.cc:237] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 990, Committed index: 990, Last appended: 1.990, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 991 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:14:31.941169 28735 raft_consensus.cc:1261] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 990. Preceding OpId from leader: term: 1 index: 991. (index mismatch)
I20240503 14:14:31.943850 31987 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 991, Last known committed idx: 989, Time since last communication: 0.000s
I20240503 14:14:31.948099 31991 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.991: config changed from index 914 to 991, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 991 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:31.952489 28735 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Committing config change with OpId 1.991: config changed from index 914 to 991, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 991 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:31.956557 27166 catalog_manager.cc:5484] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 914 to 991, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 991 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } }
I20240503 14:14:32.022692 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c65982ba3034daeacd3407a5b6877e1 with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 991) from {username='slave'} at 127.0.0.1:60240
I20240503 14:14:32.036031 32069 tablet_replica.cc:330] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:14:32.036314 32069 raft_consensus.cc:2229] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:32.036549 32069 raft_consensus.cc:2258] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:32.037715 32069 ts_tablet_manager.cc:1875] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:32.044350 32069 ts_tablet_manager.cc:1888] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.990
I20240503 14:14:32.044539 32069 log.cc:1196] T 2c65982ba3034daeacd3407a5b6877e1 P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/2c65982ba3034daeacd3407a5b6877e1
I20240503 14:14:32.045794 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet 2c65982ba3034daeacd3407a5b6877e1 (table rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]) successfully deleted
I20240503 14:14:33.242681 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:33.264298 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:33.275367 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:33.334673 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:33.349406 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:33.501082 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:33.528169 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:34.031612 27439 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 871, Committed index: 871, Last appended: 1.871, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:34.051330 28865 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 871. Preceding OpId from leader: term: 1 index: 872. (index mismatch)
I20240503 14:14:34.053274 31074 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 872, Last known committed idx: 871, Time since last communication: 0.000s
I20240503 14:14:34.072971 31991 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.872: config changed from index 296 to 872, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:34.078821 27170 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 296 to 872, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:34.083667 28865 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Committing config change with OpId 1.872: config changed from index 296 to 872, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
W20240503 14:14:34.185611 27377 consensus_peers.cc:485] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd18d7c1f6f64ef08d227e039e369cbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:34.409211 28603 consensus_queue.cc:237] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1306, Committed index: 1306, Last appended: 2.1306, Last appended by leader: 655, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1307 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:34.421770 27439 raft_consensus.cc:1261] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer c57eaae6a75a43679120a10c6d3090c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 1306. Preceding OpId from leader: term: 2 index: 1307. (index mismatch)
I20240503 14:14:34.422345 31330 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1307, Last known committed idx: 1306, Time since last communication: 0.000s
I20240503 14:14:34.426715 31330 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 2 LEADER]: Committing config change with OpId 2.1307: config changed from index 1071 to 1307, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 1307 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:34.427129 27439 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Committing config change with OpId 2.1307: config changed from index 1071 to 1307, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New config: { opid_index: 1307 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:34.432344 27166 catalog_manager.cc:5484] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 reported cstate change: config changed from index 1071 to 1307, VOTER 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6) added. New cstate: current_term: 2 leader_uuid: "c57eaae6a75a43679120a10c6d3090c7" committed_config { opid_index: 1307 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
W20240503 14:14:34.439473 28538 consensus_peers.cc:485] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 -> Peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729): Couldn't send request to peer 01c7323dc6f84fe0bc5eca4fcad8f078. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3923478811a4a0f82ec9f013ef7735d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:34.605377 32191 ts_tablet_manager.cc:902] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:14:34.799288 32191 tablet_copy_client.cc:308] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:14:34.799983 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet dd18d7c1f6f64ef08d227e039e369cbc from peer c57eaae6a75a43679120a10c6d3090c7 ({username='slave'} at 127.20.99.4:49171)
I20240503 14:14:34.800132 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet dd18d7c1f6f64ef08d227e039e369cbc from peer c57eaae6a75a43679120a10c6d3090c7 at {username='slave'} at 127.20.99.4:49171: session id = c57eaae6a75a43679120a10c6d3090c7-dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:14:34.802757 27462 tablet_copy_source_session.cc:205] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:34.806133 32191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd18d7c1f6f64ef08d227e039e369cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.816264 32191 tablet_copy_client.cc:783] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:34.816635 32191 tablet_copy_client.cc:647] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:34.823135 32191 tablet_copy_client.cc:523] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:34.829358 32191 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Bootstrap starting.
I20240503 14:14:34.931061 27440 consensus_queue.cc:237] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1044, Committed index: 1044, Last appended: 1.1045, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
W20240503 14:14:34.933125 27377 consensus_peers.cc:485] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c65982ba3034daeacd3407a5b6877e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:34.935489 28735 raft_consensus.cc:1261] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1045. Preceding OpId from leader: term: 1 index: 1046. (index mismatch)
I20240503 14:14:34.944672 32132 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1046, Last known committed idx: 1044, Time since last communication: 0.000s
I20240503 14:14:34.992265 31991 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.1046: config changed from index 991 to 1046, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:35.008138 27166 catalog_manager.cc:5484] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 991 to 1046, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:35.077752 28734 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Committing config change with OpId 1.1046: config changed from index 991 to 1046, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) added. New config: { opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:35.218277 32216 ts_tablet_manager.cc:902] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Initiating tablet copy from peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243)
I20240503 14:14:35.309629 32216 tablet_copy_client.cc:308] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:44243
I20240503 14:14:35.386348 28624 tablet_copy_service.cc:140] P c57eaae6a75a43679120a10c6d3090c7: Received BeginTabletCopySession request for tablet f3923478811a4a0f82ec9f013ef7735d from peer 01c7323dc6f84fe0bc5eca4fcad8f078 ({username='slave'} at 127.20.99.6:33689)
I20240503 14:14:35.386543 28624 tablet_copy_service.cc:161] P c57eaae6a75a43679120a10c6d3090c7: Beginning new tablet copy session on tablet f3923478811a4a0f82ec9f013ef7735d from peer 01c7323dc6f84fe0bc5eca4fcad8f078 at {username='slave'} at 127.20.99.6:33689: session id = 01c7323dc6f84fe0bc5eca4fcad8f078-f3923478811a4a0f82ec9f013ef7735d
I20240503 14:14:35.390516 28624 tablet_copy_source_session.cc:205] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:35.442847 32216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3923478811a4a0f82ec9f013ef7735d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:35.456190 32242 ts_tablet_manager.cc:902] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:14:35.460247 32216 tablet_copy_client.cc:783] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:35.460601 32216 tablet_copy_client.cc:647] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:35.469703 32216 tablet_copy_client.cc:523] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:35.474895 32216 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap starting.
I20240503 14:14:35.490468 32242 tablet_copy_client.cc:308] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:14:35.491214 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet 2c65982ba3034daeacd3407a5b6877e1 from peer c57eaae6a75a43679120a10c6d3090c7 ({username='slave'} at 127.20.99.4:49171)
I20240503 14:14:35.491353 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet 2c65982ba3034daeacd3407a5b6877e1 from peer c57eaae6a75a43679120a10c6d3090c7 at {username='slave'} at 127.20.99.4:49171: session id = c57eaae6a75a43679120a10c6d3090c7-2c65982ba3034daeacd3407a5b6877e1
I20240503 14:14:35.494303 27462 tablet_copy_source_session.cc:205] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:35.499635 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c65982ba3034daeacd3407a5b6877e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:35.537400 32242 tablet_copy_client.cc:783] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:35.537976 32242 tablet_copy_client.cc:647] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:35.555964 32242 tablet_copy_client.cc:523] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:35.560490 32242 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap starting.
I20240503 14:14:36.029158 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:36.177079 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:36.225620 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:36.389353 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:14:36.401818 32191 log.cc:924] Time spent T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Append to log took a long time: real 0.071s user 0.003s sys 0.000s
I20240503 14:14:36.419271 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:36.428447 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:36.521524 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:14:36.889370 27377 consensus_peers.cc:485] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:37.118909 27505 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 142938us
W20240503 14:14:37.209950 27505 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 79053.6us
W20240503 14:14:37.989259 28538 consensus_peers.cc:485] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 -> Peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729): Couldn't send request to peer 01c7323dc6f84fe0bc5eca4fcad8f078. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:38.030278 27375 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 93569.8us
W20240503 14:14:38.035117 27377 consensus_peers.cc:485] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:38.534106 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:38.684922 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:38.778828 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:38.793736 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:38.900359 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:38.955862 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:39.121757 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:39.162717 32191 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=877 overwritten=0 applied=877 ignored=0} inserts{seen=1547 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:39.169248 32191 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Bootstrap complete.
I20240503 14:14:39.169734 32191 ts_tablet_manager.cc:1367] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Time spent bootstrapping tablet: real 4.340s user 1.025s sys 0.125s
I20240503 14:14:39.171890 32191 raft_consensus.cc:348] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:39.173684 32191 raft_consensus.cc:729] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57eaae6a75a43679120a10c6d3090c7, State: Initialized, Role: FOLLOWER
I20240503 14:14:39.179462 32191 consensus_queue.cc:260] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 877, Last appended: 1.877, Last appended by leader: 877, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:39.180143 32191 ts_tablet_manager.cc:1398] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 14:14:39.182410 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session c57eaae6a75a43679120a10c6d3090c7-dd18d7c1f6f64ef08d227e039e369cbc received from {username='slave'} at 127.20.99.4:49171
I20240503 14:14:39.182534 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session c57eaae6a75a43679120a10c6d3090c7-dd18d7c1f6f64ef08d227e039e369cbc on tablet dd18d7c1f6f64ef08d227e039e369cbc with peer c57eaae6a75a43679120a10c6d3090c7
W20240503 14:14:39.375847 27246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 147703us
I20240503 14:14:39.695840 28604 raft_consensus.cc:1203] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.871->[1.872-1.945] Dedup: 1.877->[1.878-1.945]
W20240503 14:14:40.026455 32242 log.cc:924] Time spent T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Append to log took a long time: real 0.079s user 0.004s sys 0.000s
W20240503 14:14:40.398407 28538 consensus_peers.cc:485] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 -> Peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729): Couldn't send request to peer 01c7323dc6f84fe0bc5eca4fcad8f078. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:40.589131 27377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 350639us
W20240503 14:14:40.614707 27377 consensus_peers.cc:485] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b -> Peer c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): Couldn't send request to peer c57eaae6a75a43679120a10c6d3090c7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:40.638835 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:14:40.645737 32216 log.cc:924] Time spent T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20240503 14:14:40.713109 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.728822 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:40.801496 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:40.818571 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:40.861408 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:40.872786 32242 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=1048 overwritten=0 applied=1048 ignored=0} inserts{seen=1731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:40.881793 32242 tablet_bootstrap.cc:492] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Bootstrap complete.
I20240503 14:14:40.882356 32242 ts_tablet_manager.cc:1367] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Time spent bootstrapping tablet: real 5.322s user 1.221s sys 0.106s
I20240503 14:14:40.883262 32242 raft_consensus.cc:348] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:40.883620 32242 raft_consensus.cc:729] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57eaae6a75a43679120a10c6d3090c7, State: Initialized, Role: FOLLOWER
I20240503 14:14:40.883953 32242 consensus_queue.cc:260] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1048, Last appended: 1.1048, Last appended by leader: 1048, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:40.885430 32242 ts_tablet_manager.cc:1398] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:14:40.886534 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session c57eaae6a75a43679120a10c6d3090c7-2c65982ba3034daeacd3407a5b6877e1 received from {username='slave'} at 127.20.99.4:49171
I20240503 14:14:40.886651 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session c57eaae6a75a43679120a10c6d3090c7-2c65982ba3034daeacd3407a5b6877e1 on tablet 2c65982ba3034daeacd3407a5b6877e1 with peer c57eaae6a75a43679120a10c6d3090c7
I20240503 14:14:40.899580 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:40.973879 28602 raft_consensus.cc:1203] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1045->[1.1046-1.1115] Dedup: 1.1048->[1.1049-1.1115]
I20240503 14:14:41.101996 32216 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap replayed 1/1 log segments. Stats: ops{read=1334 overwritten=0 applied=1333 ignored=0} inserts{seen=2015 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:14:41.102674 32216 tablet_bootstrap.cc:492] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Bootstrap complete.
I20240503 14:14:41.103139 32216 ts_tablet_manager.cc:1367] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent bootstrapping tablet: real 5.629s user 1.529s sys 0.115s
I20240503 14:14:41.103878 32216 raft_consensus.cc:348] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 1307 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:41.104594 32216 raft_consensus.cc:729] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c7323dc6f84fe0bc5eca4fcad8f078, State: Initialized, Role: FOLLOWER
I20240503 14:14:41.104874 32216 consensus_queue.cc:260] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1333, Last appended: 2.1334, Last appended by leader: 1334, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1307 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:41.105542 32216 ts_tablet_manager.cc:1398] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:41.108520 28624 tablet_copy_service.cc:342] P c57eaae6a75a43679120a10c6d3090c7: Request end of tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-f3923478811a4a0f82ec9f013ef7735d received from {username='slave'} at 127.20.99.6:33689
I20240503 14:14:41.108678 28624 tablet_copy_service.cc:433] P c57eaae6a75a43679120a10c6d3090c7: ending tablet copy session 01c7323dc6f84fe0bc5eca4fcad8f078-f3923478811a4a0f82ec9f013ef7735d on tablet f3923478811a4a0f82ec9f013ef7735d with peer 01c7323dc6f84fe0bc5eca4fcad8f078
I20240503 14:14:41.292395 28863 raft_consensus.cc:1203] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1306->[2.1307-2.1440] Dedup: 2.1334->[2.1335-2.1440]
I20240503 14:14:42.204442 28845 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:14:42.263514 28929 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:42553, user_credentials={real_user=slave}} blocked reactor thread for 75008.4us
I20240503 14:14:42.281276 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:42.343770 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:42.347870 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:42.368116 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:42.449131 27420 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:14:42.550729 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:42.980392 27442 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc"
dest_uuid: "aad738f0c10848789c2c811160990f8b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48316
I20240503 14:14:42.980607 27442 raft_consensus.cc:595] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:43.060994 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.067235 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.073832 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.081373 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.096798 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.113013 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.130939 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.144551 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.159068 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.183532 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.199319 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.212873 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.228613 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.246125 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.266463 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.284436 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.305402 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.328843 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
I20240503 14:14:43.334064 32527 raft_consensus.cc:979] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: : Instructing follower 01c7323dc6f84fe0bc5eca4fcad8f078 to start an election
I20240503 14:14:43.334214 32527 raft_consensus.cc:1067] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Signalling peer 01c7323dc6f84fe0bc5eca4fcad8f078 to start an election
I20240503 14:14:43.337986 28865 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc"
dest_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078"
from {username='slave'} at 127.20.99.2:45927
I20240503 14:14:43.338184 28865 raft_consensus.cc:482] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:43.338268 28865 raft_consensus.cc:3037] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:43.341081 28865 raft_consensus.cc:504] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
W20240503 14:14:43.349193 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
W20240503 14:14:43.371484 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Service unavailable: leader transfer in progress
I20240503 14:14:43.373529 28604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc" candidate_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: true dest_uuid: "c57eaae6a75a43679120a10c6d3090c7"
I20240503 14:14:43.373916 28604 raft_consensus.cc:3037] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:43.375106 28865 leader_election.cc:288] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 2 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553), c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243)
I20240503 14:14:43.379021 28604 raft_consensus.cc:2454] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c7323dc6f84fe0bc5eca4fcad8f078 in term 2.
I20240503 14:14:43.379797 28800 leader_election.cc:302] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c7323dc6f84fe0bc5eca4fcad8f078, c57eaae6a75a43679120a10c6d3090c7; no voters:
I20240503 14:14:43.387241 32556 raft_consensus.cc:2781] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:43.388594 27440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd18d7c1f6f64ef08d227e039e369cbc" candidate_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" candidate_term: 2 candidate_status { last_received { term: 1 index: 999 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:14:43.388883 27440 raft_consensus.cc:3032] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:43.388983 27440 raft_consensus.cc:729] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:14:43.389322 27440 consensus_queue.cc:260] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 999, Committed index: 999, Last appended: 1.999, Last appended by leader: 999, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:43.389793 27440 raft_consensus.cc:3037] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:43.390412 32556 raft_consensus.cc:686] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 LEADER]: Becoming Leader. State: Replica: 01c7323dc6f84fe0bc5eca4fcad8f078, State: Running, Role: LEADER
I20240503 14:14:43.390740 32556 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 999, Committed index: 999, Last appended: 1.999, Last appended by leader: 999, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:43.392720 27440 raft_consensus.cc:2454] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c7323dc6f84fe0bc5eca4fcad8f078 in term 2.
W20240503 14:14:43.396550 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38724: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:14:43.402036 27166 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 reported cstate change: term changed from 1 to 2, leader changed from aad738f0c10848789c2c811160990f8b (127.20.99.2) to 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6). New cstate: current_term: 2 leader_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" committed_config { opid_index: 872 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:43.692365 28603 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 1 index: 999. Preceding OpId from leader: term: 2 index: 1001. (index mismatch)
I20240503 14:14:43.694304 32556 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1000, Last known committed idx: 999, Time since last communication: 0.000s
I20240503 14:14:43.695278 27440 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 1 index: 999. Preceding OpId from leader: term: 2 index: 1001. (index mismatch)
I20240503 14:14:43.700408 32556 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1000, Last known committed idx: 999, Time since last communication: 0.000s
I20240503 14:14:43.737157 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:43.767004 27420 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:43.769007 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:43.823309 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:43.846472 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:43.955878 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:43.998845 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:44.657305 28865 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1020, Committed index: 1020, Last appended: 2.1020, Last appended by leader: 999, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1021 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:44.659119 28602 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 2 index: 1020. Preceding OpId from leader: term: 2 index: 1021. (index mismatch)
I20240503 14:14:44.659868 32558 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1021, Last known committed idx: 1020, Time since last communication: 0.000s
I20240503 14:14:44.666507 32558 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 LEADER]: Committing config change with OpId 2.1021: config changed from index 872 to 1021, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1021 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:44.673089 28601 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Committing config change with OpId 2.1021: config changed from index 872 to 1021, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1021 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:44.682672 27162 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 reported cstate change: config changed from index 872 to 1021, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" committed_config { opid_index: 1021 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
W20240503 14:14:44.814251 27507 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 43641.2us
I20240503 14:14:44.886263 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet dd18d7c1f6f64ef08d227e039e369cbc with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 1021) from {username='slave'} at 127.0.0.1:50958
I20240503 14:14:44.926270 32642 tablet_replica.cc:330] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:14:44.929880 32642 raft_consensus.cc:2229] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:44.930402 32642 raft_consensus.cc:2258] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:44.931880 32642 ts_tablet_manager.cc:1875] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:44.946964 32642 ts_tablet_manager.cc:1888] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1020
I20240503 14:14:44.948222 32642 log.cc:1196] T dd18d7c1f6f64ef08d227e039e369cbc P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:14:44.950963 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet dd18d7c1f6f64ef08d227e039e369cbc (table rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]) successfully deleted
I20240503 14:14:46.031821 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:46.090402 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:46.271682 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:46.299971 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:46.313856 27420 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:14:46.349454 28715 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:14:46.377396 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:46.749370 27439 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c65982ba3034daeacd3407a5b6877e1"
dest_uuid: "aad738f0c10848789c2c811160990f8b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53560
I20240503 14:14:46.749567 27439 raft_consensus.cc:595] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Received request to transfer leadership
W20240503 14:14:46.845360 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.851352 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.859930 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.872133 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.882695 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.889686 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.901113 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.914904 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.928242 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.941278 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.964553 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.980723 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:46.995733 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:47.013979 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:47.030587 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:47.050719 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:47.069094 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:47.088997 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
W20240503 14:14:47.117825 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:47.139704 32729 raft_consensus.cc:979] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: : Instructing follower d7a1a0df5dd047ddbacafd0870ed01d1 to start an election
I20240503 14:14:47.139871 32729 raft_consensus.cc:1067] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Signalling peer d7a1a0df5dd047ddbacafd0870ed01d1 to start an election
I20240503 14:14:47.142062 28731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c65982ba3034daeacd3407a5b6877e1"
dest_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
from {username='slave'} at 127.20.99.2:39549
W20240503 14:14:47.142179 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Service unavailable: leader transfer in progress
I20240503 14:14:47.142365 28731 raft_consensus.cc:482] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:47.142464 28731 raft_consensus.cc:3037] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.146222 28731 raft_consensus.cc:504] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:47.148880 28731 leader_election.cc:288] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [CANDIDATE]: Term 2 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553), c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243)
I20240503 14:14:47.149121 27439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c65982ba3034daeacd3407a5b6877e1" candidate_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1244 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:14:47.149348 27439 raft_consensus.cc:3032] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:47.149436 27439 raft_consensus.cc:729] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:14:47.149705 27439 consensus_queue.cc:260] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1244, Committed index: 1244, Last appended: 1.1244, Last appended by leader: 1244, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:47.150079 27439 raft_consensus.cc:3037] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.153888 27439 raft_consensus.cc:2454] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7a1a0df5dd047ddbacafd0870ed01d1 in term 2.
I20240503 14:14:47.154933 28669 leader_election.cc:302] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad738f0c10848789c2c811160990f8b, d7a1a0df5dd047ddbacafd0870ed01d1; no voters:
W20240503 14:14:47.165779 27398 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:14:47.188097 32745 raft_consensus.cc:2781] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:47.205008 32745 raft_consensus.cc:686] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 LEADER]: Becoming Leader. State: Replica: d7a1a0df5dd047ddbacafd0870ed01d1, State: Running, Role: LEADER
I20240503 14:14:47.205953 32745 consensus_queue.cc:237] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1244, Committed index: 1244, Last appended: 1.1244, Last appended by leader: 1244, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:47.214609 27162 catalog_manager.cc:5484] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 reported cstate change: term changed from 1 to 2, leader changed from aad738f0c10848789c2c811160990f8b (127.20.99.2) to d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5). New cstate: current_term: 2 leader_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" committed_config { opid_index: 1046 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
W20240503 14:14:47.239523 27400 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38734: Illegal state: replica aad738f0c10848789c2c811160990f8b is not leader of this config: current role FOLLOWER
I20240503 14:14:47.428719 27439 raft_consensus.cc:1261] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer d7a1a0df5dd047ddbacafd0870ed01d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1244. Preceding OpId from leader: term: 2 index: 1246. (index mismatch)
I20240503 14:14:47.429924 32746 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1245, Last known committed idx: 1244, Time since last communication: 0.000s
I20240503 14:14:47.434420 32600 mvcc.cc:205] Tried to move back new op lower bound from 7023598335697330176 to 7023598334813282304. Current Snapshot: MvccSnapshot[applied={T|T < 7023598335697330176}]
I20240503 14:14:47.476434 28601 raft_consensus.cc:3037] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:47.476595 28603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c65982ba3034daeacd3407a5b6877e1" candidate_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1244 } } ignore_live_leader: true dest_uuid: "c57eaae6a75a43679120a10c6d3090c7"
I20240503 14:14:47.479542 28601 raft_consensus.cc:1261] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Refusing update from remote peer d7a1a0df5dd047ddbacafd0870ed01d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1244. Preceding OpId from leader: term: 2 index: 1246. (index mismatch)
I20240503 14:14:47.480141 32745 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1245, Last known committed idx: 1244, Time since last communication: 0.000s
I20240503 14:14:47.869091 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:47.884076 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:48.037983 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:14:48.173007 28800 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:44243, user_credentials={real_user=slave}} blocked reactor thread for 62334.1us
I20240503 14:14:48.256485 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:48.262579 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:48.278591 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:48.398582 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:49.294994 28731 consensus_queue.cc:237] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1281, Committed index: 1281, Last appended: 2.1281, Last appended by leader: 1244, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1282 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:14:49.296573 28603 raft_consensus.cc:1261] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Refusing update from remote peer d7a1a0df5dd047ddbacafd0870ed01d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 1281. Preceding OpId from leader: term: 2 index: 1282. (index mismatch)
I20240503 14:14:49.297088 32746 consensus_queue.cc:1035] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1282, Last known committed idx: 1281, Time since last communication: 0.000s
I20240503 14:14:49.299513 32746 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 LEADER]: Committing config change with OpId 2.1282: config changed from index 1046 to 1282, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1282 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:49.306739 27162 catalog_manager.cc:5484] T 2c65982ba3034daeacd3407a5b6877e1 P d7a1a0df5dd047ddbacafd0870ed01d1 reported cstate change: config changed from index 1046 to 1282, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" committed_config { opid_index: 1282 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:49.310899 28603 raft_consensus.cc:2932] T 2c65982ba3034daeacd3407a5b6877e1 P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Committing config change with OpId 2.1282: config changed from index 1046 to 1282, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1282 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } }
I20240503 14:14:49.415545 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c65982ba3034daeacd3407a5b6877e1 with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 1282) from {username='slave'} at 127.0.0.1:50958
I20240503 14:14:49.561813 378 tablet_replica.cc:330] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:14:49.562141 378 raft_consensus.cc:2229] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.567333 378 raft_consensus.cc:2258] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.571293 378 ts_tablet_manager.cc:1875] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.578590 378 ts_tablet_manager.cc:1888] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1281
I20240503 14:14:49.578784 378 log.cc:1196] T 2c65982ba3034daeacd3407a5b6877e1 P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/2c65982ba3034daeacd3407a5b6877e1
I20240503 14:14:49.581388 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet 2c65982ba3034daeacd3407a5b6877e1 (table rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]) successfully deleted
I20240503 14:14:50.441615 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:50.481696 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:50.512166 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:50.573961 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:50.625347 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:50.691582 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:50.722177 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:51.342018 28602 consensus_queue.cc:237] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1646, Committed index: 1646, Last appended: 2.1646, Last appended by leader: 655, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1647 OBSOLETE_local: false peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:14:51.360632 28865 raft_consensus.cc:1261] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Refusing update from remote peer c57eaae6a75a43679120a10c6d3090c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 1646. Preceding OpId from leader: term: 2 index: 1647. (index mismatch)
I20240503 14:14:51.378086 32722 consensus_queue.cc:1035] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1647, Last known committed idx: 1646, Time since last communication: 0.000s
I20240503 14:14:51.422029 32722 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 [term 2 LEADER]: Committing config change with OpId 2.1647: config changed from index 1307 to 1647, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1647 OBSOLETE_local: false peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:51.425822 28865 raft_consensus.cc:2932] T f3923478811a4a0f82ec9f013ef7735d P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Committing config change with OpId 2.1647: config changed from index 1307 to 1647, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1647 OBSOLETE_local: false peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:51.428968 27170 catalog_manager.cc:5484] T f3923478811a4a0f82ec9f013ef7735d P c57eaae6a75a43679120a10c6d3090c7 reported cstate change: config changed from index 1307 to 1647, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "c57eaae6a75a43679120a10c6d3090c7" committed_config { opid_index: 1647 OBSOLETE_local: false peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } }
I20240503 14:14:51.472431 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet f3923478811a4a0f82ec9f013ef7735d with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 1647) from {username='slave'} at 127.0.0.1:50958
I20240503 14:14:51.478087 487 tablet_replica.cc:330] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:14:51.478663 487 raft_consensus.cc:2229] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:51.479096 487 raft_consensus.cc:2258] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:51.480683 487 ts_tablet_manager.cc:1875] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:51.492211 487 ts_tablet_manager.cc:1888] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1646
I20240503 14:14:51.492529 487 log.cc:1196] T f3923478811a4a0f82ec9f013ef7735d P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/f3923478811a4a0f82ec9f013ef7735d
I20240503 14:14:51.494436 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet f3923478811a4a0f82ec9f013ef7735d (table rebalance_test_table_0 [id=e54100be49ef49f9a7865aac4c1988e7]) successfully deleted
W20240503 14:14:52.343371 27505 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:45199, user_credentials={real_user=slave}} blocked reactor thread for 80360.2us
I20240503 14:14:52.620568 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:52.637001 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:52.646171 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:52.651163 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:52.682338 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:52.684773 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:52.692797 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:53.749645 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:53.801149 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:53.810142 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:53.833878 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:53.888859 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:53.928205 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:53.941171 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:14:54.093811 28801 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:43621, user_credentials={real_user=slave}} blocked reactor thread for 44325.7us
I20240503 14:14:56.983073 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:57.167974 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:57.264674 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:57.283442 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:57.321801 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:57.402002 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:57.435356 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:58.984788 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:59.031248 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:59.050886 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:59.071038 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:59.073485 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:59.087561 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:59.147408 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:59.866340 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:59.897716 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:14:59.898764 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:14:59.903015 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:59.922315 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:14:59.933284 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:14:59.953258 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
16e0966be88a4fdfbedfd01d9ae02384 | 127.20.99.62:38501 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.99.20:46773 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
01c7323dc6f84fe0bc5eca4fcad8f078 | 127.20.99.6:45729 | HEALTHY | <none> | 2 | 0
2116de1ecc2f48daa3c8cca7edc82a95 | 127.20.99.3:43621 | HEALTHY | <none> | 3 | 0
60da896b8369489994355cc0e9050514 | 127.20.99.1:45565 | HEALTHY | <none> | 3 | 0
aad738f0c10848789c2c811160990f8b | 127.20.99.2:42553 | HEALTHY | <none> | 4 | 0
c57eaae6a75a43679120a10c6d3090c7 | 127.20.99.4:44243 | HEALTHY | <none> | 1 | 0
d7a1a0df5dd047ddbacafd0870ed01d1 | 127.20.99.5:37731 | HEALTHY | <none> | 1 | 0
dac9fd6f6a404989ae1a460ac3cde0fe | 127.20.99.7:45199 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.20.99.1 | experimental | 127.20.99.1:45565
local_ip_for_outbound_sockets | 127.20.99.2 | experimental | 127.20.99.2:42553
local_ip_for_outbound_sockets | 127.20.99.3 | experimental | 127.20.99.3:43621
local_ip_for_outbound_sockets | 127.20.99.4 | experimental | 127.20.99.4:44243
local_ip_for_outbound_sockets | 127.20.99.5 | experimental | 127.20.99.5:37731
local_ip_for_outbound_sockets | 127.20.99.6 | experimental | 127.20.99.6:45729
local_ip_for_outbound_sockets | 127.20.99.7 | experimental | 127.20.99.7:45199
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.99.1:45565
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.99.2:42553
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.99.3:43621
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.99.4:44243
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.99.5:37731
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.99.6:45729
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.99.7:45199
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.99.20:46773 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:15:00.011991 20876 log_verifier.cc:126] Checking tablet 244c146eb7624b3585068f9c2de0c4ed
I20240503 14:15:00.568909 20876 log_verifier.cc:177] Verified matching terms for 1514 ops in tablet 244c146eb7624b3585068f9c2de0c4ed
I20240503 14:15:00.570744 20876 log_verifier.cc:126] Checking tablet 2c65982ba3034daeacd3407a5b6877e1
I20240503 14:15:01.055413 20876 log_verifier.cc:177] Verified matching terms for 1469 ops in tablet 2c65982ba3034daeacd3407a5b6877e1
I20240503 14:15:01.057435 20876 log_verifier.cc:126] Checking tablet 4180d30f1d3449ff9c28c69599098ff4
I20240503 14:15:01.520867 20876 log_verifier.cc:177] Verified matching terms for 1409 ops in tablet 4180d30f1d3449ff9c28c69599098ff4
I20240503 14:15:01.523061 20876 log_verifier.cc:126] Checking tablet 44eb2dd4af5444f2a6a75b50bce0e03f
I20240503 14:15:01.539336 20876 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44eb2dd4af5444f2a6a75b50bce0e03f
I20240503 14:15:01.539460 20876 log_verifier.cc:126] Checking tablet 54fbf7658e87471895a21e486f59352a
I20240503 14:15:02.158315 20876 log_verifier.cc:177] Verified matching terms for 1842 ops in tablet 54fbf7658e87471895a21e486f59352a
I20240503 14:15:02.161818 20876 log_verifier.cc:126] Checking tablet 63dc99c47a8a4f379b3349837d4b7264
I20240503 14:15:02.685930 20876 log_verifier.cc:177] Verified matching terms for 1591 ops in tablet 63dc99c47a8a4f379b3349837d4b7264
I20240503 14:15:02.687582 20876 log_verifier.cc:126] Checking tablet 86890b3c77ba41d7b08579defb20aff1
I20240503 14:15:03.124905 20876 log_verifier.cc:177] Verified matching terms for 1249 ops in tablet 86890b3c77ba41d7b08579defb20aff1
I20240503 14:15:03.126155 20876 log_verifier.cc:126] Checking tablet 98bec1722172403aa21b3d060ed636d2
I20240503 14:15:03.625254 20876 log_verifier.cc:177] Verified matching terms for 1478 ops in tablet 98bec1722172403aa21b3d060ed636d2
I20240503 14:15:03.626813 20876 log_verifier.cc:126] Checking tablet a352cdeffa9a4b85badcefa545885bc3
I20240503 14:15:04.160372 20876 log_verifier.cc:177] Verified matching terms for 1541 ops in tablet a352cdeffa9a4b85badcefa545885bc3
I20240503 14:15:04.163343 20876 log_verifier.cc:126] Checking tablet ae02587deb4a450da3171f30a1501950
I20240503 14:15:04.575505 20876 log_verifier.cc:177] Verified matching terms for 1251 ops in tablet ae02587deb4a450da3171f30a1501950
I20240503 14:15:04.576836 20876 log_verifier.cc:126] Checking tablet b44e3d7641cc42fdb789fbfd0405f072
I20240503 14:15:05.157888 20876 log_verifier.cc:177] Verified matching terms for 1706 ops in tablet b44e3d7641cc42fdb789fbfd0405f072
I20240503 14:15:05.159729 20876 log_verifier.cc:126] Checking tablet c723f0969bd9472d833bdbb54ee67556
I20240503 14:15:05.686853 20876 log_verifier.cc:177] Verified matching terms for 1476 ops in tablet c723f0969bd9472d833bdbb54ee67556
I20240503 14:15:05.688426 20876 log_verifier.cc:126] Checking tablet dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:15:06.092537 20876 log_verifier.cc:177] Verified matching terms for 1273 ops in tablet dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:15:06.094257 20876 log_verifier.cc:126] Checking tablet e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:15:06.587206 20876 log_verifier.cc:177] Verified matching terms for 1427 ops in tablet e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:15:06.588843 20876 log_verifier.cc:126] Checking tablet f3923478811a4a0f82ec9f013ef7735d
I20240503 14:15:07.195613 20876 log_verifier.cc:177] Verified matching terms for 1814 ops in tablet f3923478811a4a0f82ec9f013ef7735d
I20240503 14:15:07.197564 20876 log_verifier.cc:126] Checking tablet f576c9fe2ddf4494b51d41d64d18a1eb
I20240503 14:15:07.676904 20876 log_verifier.cc:177] Verified matching terms for 1463 ops in tablet f576c9fe2ddf4494b51d41d64d18a1eb
I20240503 14:15:07.690939 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.8:0
--local_ip_for_outbound_sockets=127.20.99.8
--webserver_interface=127.20.99.8
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--builtin_ntp_servers=127.20.99.20:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:15:08.068768 853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:08.069646 853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:08.069804 853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:08.077432 853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:08.077560 853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:08.077744 853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.8
I20240503 14:15:08.087740 853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.99.8
--webserver_port=0
--tserver_master_addrs=127.20.99.62:38501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.8
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:08.089128 853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:08.090875 853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:08.103830 860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:08.104549 862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:08.105394 859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:08.106083 853 server_base.cc:1034] running on GCE node
I20240503 14:15:08.106848 853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:08.107908 853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:08.109597 853 hybrid_clock.cc:648] HybridClock initialized: now 1714745708109058 us; error 46 us; skew 500 ppm
I20240503 14:15:08.109922 853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:08.112174 853 webserver.cc:460] Webserver started at http://127.20.99.8:45719/ using document root <none> and password file <none>
I20240503 14:15:08.112954 853 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:08.113052 853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:08.113389 853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:08.116070 853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "159f56e814134c3db8af6f1760255630"
format_stamp: "Formatted at 2024-05-03 14:15:08 on dist-test-slave-p0gh"
I20240503 14:15:08.116840 853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "159f56e814134c3db8af6f1760255630"
format_stamp: "Formatted at 2024-05-03 14:15:08 on dist-test-slave-p0gh"
I20240503 14:15:08.122159 853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240503 14:15:08.125846 868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:08.127306 853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 14:15:08.127489 853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "159f56e814134c3db8af6f1760255630"
format_stamp: "Formatted at 2024-05-03 14:15:08 on dist-test-slave-p0gh"
I20240503 14:15:08.127707 853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:08.174502 853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:08.175583 853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:08.175891 853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:08.177057 853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:08.178916 853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:08.179028 853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:08.179131 853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:08.179219 853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:08.218489 853 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.8:32857
I20240503 14:15:08.218544 980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.8:32857 every 8 connection(s)
I20240503 14:15:08.220007 853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:15:08.225998 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 853
I20240503 14:15:08.226215 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:15:08.243268 981 heartbeater.cc:347] Connected to a master server at 127.20.99.62:38501
I20240503 14:15:08.243695 981 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:08.244827 981 heartbeater.cc:511] Master 127.20.99.62:38501 requested a full tablet report, sending...
I20240503 14:15:08.246984 27166 ts_manager.cc:194] Registered new tserver with Master: 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857)
I20240503 14:15:08.247866 27166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.8:33663
I20240503 14:15:08.790194 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:08.790587 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:08.808979 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:08.823302 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:08.827481 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:08.829952 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:08.844419 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:08.871073 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.061031 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.072316 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:09.080518 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:09.090370 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:09.090930 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.108145 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.132509 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.151654 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:09.250622 981 heartbeater.cc:503] Master 127.20.99.62:38501 was elected leader, sending a full tablet report...
I20240503 14:15:09.258756 28863 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1273, Committed index: 1273, Last appended: 2.1273, Last appended by leader: 999, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1274 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:09.261166 28600 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 2 index: 1273. Preceding OpId from leader: term: 2 index: 1274. (index mismatch)
I20240503 14:15:09.261742 833 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1274, Last known committed idx: 1273, Time since last communication: 0.000s
I20240503 14:15:09.264441 1087 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 LEADER]: Committing config change with OpId 2.1274: config changed from index 1021 to 1274, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1274 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
W20240503 14:15:09.268440 28801 consensus_peers.cc:485] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd18d7c1f6f64ef08d227e039e369cbc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.268424 28600 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Committing config change with OpId 2.1274: config changed from index 1021 to 1274, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1274 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.268846 27165 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 reported cstate change: config changed from index 1021 to 1274, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New cstate: current_term: 2 leader_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" committed_config { opid_index: 1274 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.309281 27571 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1427, Committed index: 1427, Last appended: 2.1427, Last appended by leader: 565, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:09.311673 28995 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Refusing update from remote peer 2116de1ecc2f48daa3c8cca7edc82a95: Log matching property violated. Preceding OpId in replica: term: 2 index: 1427. Preceding OpId from leader: term: 2 index: 1428. (index mismatch)
I20240503 14:15:09.312327 838 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1428, Last known committed idx: 1427, Time since last communication: 0.000s
I20240503 14:15:09.319983 1110 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Committing config change with OpId 2.1428: config changed from index 580 to 1428, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.320648 28995 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Committing config change with OpId 2.1428: config changed from index 580 to 1428, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
W20240503 14:15:09.325316 27508 consensus_peers.cc:485] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8102c1d08074ea7bb5be6c5645d1fcd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.325356 27167 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 580 to 1428, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New cstate: current_term: 2 leader_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" committed_config { opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.356928 27439 consensus_queue.cc:237] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1478, Committed index: 1478, Last appended: 1.1478, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:09.359373 28734 raft_consensus.cc:1261] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Refusing update from remote peer aad738f0c10848789c2c811160990f8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1478. Preceding OpId from leader: term: 1 index: 1479. (index mismatch)
I20240503 14:15:09.359884 1129 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1479, Last known committed idx: 1478, Time since last communication: 0.000s
I20240503 14:15:09.362898 643 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Committing config change with OpId 1.1479: config changed from index 832 to 1479, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.363487 28734 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Committing config change with OpId 1.1479: config changed from index 832 to 1479, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.368062 27165 catalog_manager.cc:5484] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b reported cstate change: config changed from index 832 to 1479, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "aad738f0c10848789c2c811160990f8b" committed_config { opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
W20240503 14:15:09.368707 27378 consensus_peers.cc:485] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98bec1722172403aa21b3d060ed636d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.396390 28995 consensus_queue.cc:237] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1541, Committed index: 1541, Last appended: 2.1541, Last appended by leader: 763, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1542 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:09.398845 27307 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 1541. Preceding OpId from leader: term: 2 index: 1542. (index mismatch)
I20240503 14:15:09.399709 1150 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1542, Last known committed idx: 1541, Time since last communication: 0.000s
I20240503 14:15:09.402335 849 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 LEADER]: Committing config change with OpId 2.1542: config changed from index 795 to 1542, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1542 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.403438 27307 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Committing config change with OpId 2.1542: config changed from index 795 to 1542, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New config: { opid_index: 1542 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
W20240503 14:15:09.405973 28931 consensus_peers.cc:485] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a352cdeffa9a4b85badcefa545885bc3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.408746 27167 catalog_manager.cc:5484] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 795 to 1542, VOTER 159f56e814134c3db8af6f1760255630 (127.20.99.8) added. New cstate: current_term: 2 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 1542 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:09.512681 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:09.539073 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.540343 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:09.542497 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.543165 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:09.550361 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.559767 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.562125 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:09.770743 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:09.787624 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:09.791885 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:09.800752 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:09.814918 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.822350 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:09.823844 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:09.832507 1229 ts_tablet_manager.cc:902] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: Initiating tablet copy from peer 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621)
I20240503 14:15:09.855955 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:09.869966 1231 ts_tablet_manager.cc:902] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: Initiating tablet copy from peer dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7:45199)
I20240503 14:15:09.880896 1232 ts_tablet_manager.cc:902] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Initiating tablet copy from peer 01c7323dc6f84fe0bc5eca4fcad8f078 (127.20.99.6:45729)
I20240503 14:15:09.906244 1232 tablet_copy_client.cc:308] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:45729
I20240503 14:15:09.923072 28885 tablet_copy_service.cc:140] P 01c7323dc6f84fe0bc5eca4fcad8f078: Received BeginTabletCopySession request for tablet dd18d7c1f6f64ef08d227e039e369cbc from peer 159f56e814134c3db8af6f1760255630 ({username='slave'} at 127.20.99.8:57413)
I20240503 14:15:09.923308 28885 tablet_copy_service.cc:161] P 01c7323dc6f84fe0bc5eca4fcad8f078: Beginning new tablet copy session on tablet dd18d7c1f6f64ef08d227e039e369cbc from peer 159f56e814134c3db8af6f1760255630 at {username='slave'} at 127.20.99.8:57413: session id = 159f56e814134c3db8af6f1760255630-dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:15:09.924901 1247 ts_tablet_manager.cc:902] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: Initiating tablet copy from peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
I20240503 14:15:09.927048 28885 tablet_copy_source_session.cc:205] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:09.932719 1231 tablet_copy_client.cc:308] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.7:45199
I20240503 14:15:09.933233 1229 tablet_copy_client.cc:308] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:43621
I20240503 14:15:09.941150 1232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd18d7c1f6f64ef08d227e039e369cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:09.948102 1247 tablet_copy_client.cc:308] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:42553
I20240503 14:15:09.954087 27592 tablet_copy_service.cc:140] P 2116de1ecc2f48daa3c8cca7edc82a95: Received BeginTabletCopySession request for tablet e8102c1d08074ea7bb5be6c5645d1fcd from peer 159f56e814134c3db8af6f1760255630 ({username='slave'} at 127.20.99.8:53637)
I20240503 14:15:09.954306 27592 tablet_copy_service.cc:161] P 2116de1ecc2f48daa3c8cca7edc82a95: Beginning new tablet copy session on tablet e8102c1d08074ea7bb5be6c5645d1fcd from peer 159f56e814134c3db8af6f1760255630 at {username='slave'} at 127.20.99.8:53637: session id = 159f56e814134c3db8af6f1760255630-e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:15:09.958523 27592 tablet_copy_source_session.cc:205] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:09.961167 1229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8102c1d08074ea7bb5be6c5645d1fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:09.964291 29015 tablet_copy_service.cc:140] P dac9fd6f6a404989ae1a460ac3cde0fe: Received BeginTabletCopySession request for tablet a352cdeffa9a4b85badcefa545885bc3 from peer 159f56e814134c3db8af6f1760255630 ({username='slave'} at 127.20.99.8:60023)
I20240503 14:15:09.964524 29015 tablet_copy_service.cc:161] P dac9fd6f6a404989ae1a460ac3cde0fe: Beginning new tablet copy session on tablet a352cdeffa9a4b85badcefa545885bc3 from peer 159f56e814134c3db8af6f1760255630 at {username='slave'} at 127.20.99.8:60023: session id = 159f56e814134c3db8af6f1760255630-a352cdeffa9a4b85badcefa545885bc3
I20240503 14:15:09.967074 29015 tablet_copy_source_session.cc:205] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:09.968487 1229 tablet_copy_client.cc:783] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:09.968750 1231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a352cdeffa9a4b85badcefa545885bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:09.968812 1229 tablet_copy_client.cc:647] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:09.970412 1232 tablet_copy_client.cc:783] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:09.970783 1232 tablet_copy_client.cc:647] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:09.974325 27462 tablet_copy_service.cc:140] P aad738f0c10848789c2c811160990f8b: Received BeginTabletCopySession request for tablet 98bec1722172403aa21b3d060ed636d2 from peer 159f56e814134c3db8af6f1760255630 ({username='slave'} at 127.20.99.8:56867)
I20240503 14:15:09.974519 27462 tablet_copy_service.cc:161] P aad738f0c10848789c2c811160990f8b: Beginning new tablet copy session on tablet 98bec1722172403aa21b3d060ed636d2 from peer 159f56e814134c3db8af6f1760255630 at {username='slave'} at 127.20.99.8:56867: session id = 159f56e814134c3db8af6f1760255630-98bec1722172403aa21b3d060ed636d2
I20240503 14:15:09.975317 1231 tablet_copy_client.cc:783] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:09.975571 1231 tablet_copy_client.cc:647] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:09.977380 27462 tablet_copy_source_session.cc:205] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:09.979130 1229 tablet_copy_client.cc:523] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:09.979382 1232 tablet_copy_client.cc:523] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:09.980588 1247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98bec1722172403aa21b3d060ed636d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:09.983464 1232 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Bootstrap starting.
I20240503 14:15:09.983868 1231 tablet_copy_client.cc:523] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:09.985078 1229 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: Bootstrap starting.
I20240503 14:15:09.987396 1231 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: Bootstrap starting.
I20240503 14:15:10.005932 1247 tablet_copy_client.cc:783] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.006523 1247 tablet_copy_client.cc:647] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.018976 1247 tablet_copy_client.cc:523] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.029064 1247 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: Bootstrap starting.
I20240503 14:15:10.347407 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:10.390002 1232 log.cc:824] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:10.468443 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:10.490283 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:10.491194 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:10.498407 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:10.517735 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:10.551683 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:10.575428 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:10.999153 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:11.018944 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:11.074760 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:11.140836 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:11.145251 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:11.191884 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:11.192286 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:11.207500 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:11.752897 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:11.778785 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:11.799599 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:11.811837 27159 maintenance_manager.cc:419] P 16e0966be88a4fdfbedfd01d9ae02384: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033387
I20240503 14:15:11.822230 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:11.861269 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:11.898428 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:11.916697 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:11.925204 27158 maintenance_manager.cc:643] P 16e0966be88a4fdfbedfd01d9ae02384: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.113s user 0.046s sys 0.035s Metrics: {"bytes_written":12790,"cfile_init":1,"dirs.queue_time_us":382,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":24902,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1592,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":25,"thread_start_us":76322,"threads_started":2,"wal-append.queue_time_us":74855}
I20240503 14:15:11.937259 27159 maintenance_manager.cc:419] P 16e0966be88a4fdfbedfd01d9ae02384: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3139 bytes on disk
I20240503 14:15:11.951890 27158 maintenance_manager.cc:643] P 16e0966be88a4fdfbedfd01d9ae02384: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 14:15:11.953006 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:15:12.203827 27508 consensus_peers.cc:485] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:12.213268 27378 consensus_peers.cc:485] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:12.220307 28931 consensus_peers.cc:485] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:12.451642 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:12.492053 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:12.526841 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:12.541707 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:12.564594 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:12.579064 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:12.621764 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:12.678570 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:15:12.712129 28801 consensus_peers.cc:485] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 -> Peer 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857): Couldn't send request to peer 159f56e814134c3db8af6f1760255630. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:12.829960 1232 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Bootstrap replayed 1/1 log segments. Stats: ops{read=1274 overwritten=0 applied=1274 ignored=0} inserts{seen=1941 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:12.830752 1232 tablet_bootstrap.cc:492] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Bootstrap complete.
I20240503 14:15:12.831524 1232 ts_tablet_manager.cc:1367] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Time spent bootstrapping tablet: real 2.848s user 1.426s sys 0.136s
I20240503 14:15:12.838523 1232 raft_consensus.cc:348] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1274 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:12.839108 1232 raft_consensus.cc:729] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159f56e814134c3db8af6f1760255630, State: Initialized, Role: FOLLOWER
I20240503 14:15:12.840106 1232 consensus_queue.cc:260] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1274, Last appended: 2.1274, Last appended by leader: 1274, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1274 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:12.851944 1232 ts_tablet_manager.cc:1398] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20240503 14:15:12.853122 28885 tablet_copy_service.cc:342] P 01c7323dc6f84fe0bc5eca4fcad8f078: Request end of tablet copy session 159f56e814134c3db8af6f1760255630-dd18d7c1f6f64ef08d227e039e369cbc received from {username='slave'} at 127.20.99.8:57413
I20240503 14:15:12.853291 28885 tablet_copy_service.cc:433] P 01c7323dc6f84fe0bc5eca4fcad8f078: ending tablet copy session 159f56e814134c3db8af6f1760255630-dd18d7c1f6f64ef08d227e039e369cbc on tablet dd18d7c1f6f64ef08d227e039e369cbc with peer 159f56e814134c3db8af6f1760255630
I20240503 14:15:12.984611 1247 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: Bootstrap replayed 1/1 log segments. Stats: ops{read=1479 overwritten=0 applied=1479 ignored=0} inserts{seen=2108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:12.985093 1247 tablet_bootstrap.cc:492] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: Bootstrap complete.
I20240503 14:15:12.985466 1247 ts_tablet_manager.cc:1367] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: Time spent bootstrapping tablet: real 2.956s user 1.645s sys 0.153s
I20240503 14:15:12.987152 1247 raft_consensus.cc:348] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:12.987426 1247 raft_consensus.cc:729] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159f56e814134c3db8af6f1760255630, State: Initialized, Role: FOLLOWER
I20240503 14:15:12.987733 1247 consensus_queue.cc:260] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1479, Last appended: 1.1479, Last appended by leader: 1479, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:12.988296 1247 ts_tablet_manager.cc:1398] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:12.990022 27462 tablet_copy_service.cc:342] P aad738f0c10848789c2c811160990f8b: Request end of tablet copy session 159f56e814134c3db8af6f1760255630-98bec1722172403aa21b3d060ed636d2 received from {username='slave'} at 127.20.99.8:56867
I20240503 14:15:12.990132 27462 tablet_copy_service.cc:433] P aad738f0c10848789c2c811160990f8b: ending tablet copy session 159f56e814134c3db8af6f1760255630-98bec1722172403aa21b3d060ed636d2 on tablet 98bec1722172403aa21b3d060ed636d2 with peer 159f56e814134c3db8af6f1760255630
I20240503 14:15:13.130692 1229 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: Bootstrap replayed 1/1 log segments. Stats: ops{read=1428 overwritten=0 applied=1428 ignored=0} inserts{seen=2056 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:13.131263 1229 tablet_bootstrap.cc:492] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: Bootstrap complete.
I20240503 14:15:13.131843 1229 ts_tablet_manager.cc:1367] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: Time spent bootstrapping tablet: real 3.147s user 1.544s sys 0.185s
I20240503 14:15:13.132622 1229 raft_consensus.cc:348] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:13.132844 1229 raft_consensus.cc:729] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159f56e814134c3db8af6f1760255630, State: Initialized, Role: FOLLOWER
I20240503 14:15:13.133087 1229 consensus_queue.cc:260] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1428, Last appended: 2.1428, Last appended by leader: 1428, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:13.133586 1229 ts_tablet_manager.cc:1398] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:13.134385 27592 tablet_copy_service.cc:342] P 2116de1ecc2f48daa3c8cca7edc82a95: Request end of tablet copy session 159f56e814134c3db8af6f1760255630-e8102c1d08074ea7bb5be6c5645d1fcd received from {username='slave'} at 127.20.99.8:53637
I20240503 14:15:13.134531 27592 tablet_copy_service.cc:433] P 2116de1ecc2f48daa3c8cca7edc82a95: ending tablet copy session 159f56e814134c3db8af6f1760255630-e8102c1d08074ea7bb5be6c5645d1fcd on tablet e8102c1d08074ea7bb5be6c5645d1fcd with peer 159f56e814134c3db8af6f1760255630
I20240503 14:15:13.156997 935 raft_consensus.cc:1203] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1478->[1.1479-1.1479] Dedup: 1.1479->[]
I20240503 14:15:13.268113 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:13.294510 1231 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: Bootstrap replayed 1/1 log segments. Stats: ops{read=1542 overwritten=0 applied=1542 ignored=0} inserts{seen=2211 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:13.294726 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:13.300547 1231 tablet_bootstrap.cc:492] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: Bootstrap complete.
I20240503 14:15:13.301522 935 raft_consensus.cc:1203] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1273->[2.1274-2.1274] Dedup: 2.1274->[]
I20240503 14:15:13.302728 1231 ts_tablet_manager.cc:1367] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: Time spent bootstrapping tablet: real 3.315s user 1.741s sys 0.151s
I20240503 14:15:13.303589 1231 raft_consensus.cc:348] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1542 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:13.303864 1231 raft_consensus.cc:729] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159f56e814134c3db8af6f1760255630, State: Initialized, Role: FOLLOWER
I20240503 14:15:13.304116 1231 consensus_queue.cc:260] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1542, Last appended: 2.1542, Last appended by leader: 1542, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1542 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:13.308277 1231 ts_tablet_manager.cc:1398] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:15:13.309182 29015 tablet_copy_service.cc:342] P dac9fd6f6a404989ae1a460ac3cde0fe: Request end of tablet copy session 159f56e814134c3db8af6f1760255630-a352cdeffa9a4b85badcefa545885bc3 received from {username='slave'} at 127.20.99.8:60023
I20240503 14:15:13.309379 29015 tablet_copy_service.cc:433] P dac9fd6f6a404989ae1a460ac3cde0fe: ending tablet copy session 159f56e814134c3db8af6f1760255630-a352cdeffa9a4b85badcefa545885bc3 on tablet a352cdeffa9a4b85badcefa545885bc3 with peer 159f56e814134c3db8af6f1760255630
I20240503 14:15:13.321378 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:13.321434 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:13.326642 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:13.348001 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:13.349030 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:13.373842 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:13.515975 936 raft_consensus.cc:1203] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1427->[2.1428-2.1428] Dedup: 2.1428->[]
I20240503 14:15:13.536651 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:13.550211 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:13.552400 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:13.564414 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:13.564709 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:13.568770 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:13.579418 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:13.587384 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:13.692260 936 raft_consensus.cc:1203] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1541->[2.1542-2.1542] Dedup: 2.1542->[]
I20240503 14:15:13.713136 28863 consensus_queue.cc:237] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1274, Committed index: 1274, Last appended: 2.1274, Last appended by leader: 999, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1275 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:13.715605 936 raft_consensus.cc:1261] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Refusing update from remote peer 01c7323dc6f84fe0bc5eca4fcad8f078: Log matching property violated. Preceding OpId in replica: term: 2 index: 1274. Preceding OpId from leader: term: 2 index: 1275. (index mismatch)
I20240503 14:15:13.716173 1087 consensus_queue.cc:1035] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1275, Last known committed idx: 1274, Time since last communication: 0.000s
I20240503 14:15:13.719934 1533 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 LEADER]: Committing config change with OpId 2.1275: config changed from index 1274 to 1275, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) evicted. New config: { opid_index: 1275 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:13.720672 936 raft_consensus.cc:2932] T dd18d7c1f6f64ef08d227e039e369cbc P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Committing config change with OpId 2.1275: config changed from index 1274 to 1275, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) evicted. New config: { opid_index: 1275 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:13.725438 27170 catalog_manager.cc:5484] T dd18d7c1f6f64ef08d227e039e369cbc P 01c7323dc6f84fe0bc5eca4fcad8f078 reported cstate change: config changed from index 1274 to 1275, VOTER c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" committed_config { opid_index: 1275 OBSOLETE_local: false peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:13.755607 28584 tablet_service.cc:1514] Processing DeleteTablet for tablet dd18d7c1f6f64ef08d227e039e369cbc with delete_type TABLET_DATA_TOMBSTONED (TS c57eaae6a75a43679120a10c6d3090c7 not found in new config with opid_index 1275) from {username='slave'} at 127.0.0.1:37490
I20240503 14:15:13.761085 1550 tablet_replica.cc:330] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: stopping tablet replica
I20240503 14:15:13.761591 1550 raft_consensus.cc:2229] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:13.763623 1550 raft_consensus.cc:2258] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:13.766072 1550 ts_tablet_manager.cc:1875] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:13.772632 1550 ts_tablet_manager.cc:1888] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1274
I20240503 14:15:13.772800 1550 log.cc:1196] T dd18d7c1f6f64ef08d227e039e369cbc P c57eaae6a75a43679120a10c6d3090c7: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:15:13.773831 27156 catalog_manager.cc:4830] TS c57eaae6a75a43679120a10c6d3090c7 (127.20.99.4:44243): tablet dd18d7c1f6f64ef08d227e039e369cbc (table rebalance_test_table_2 [id=1a271df1692246a6aa89d544c5e294b2]) successfully deleted
I20240503 14:15:13.849571 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:13.872538 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:13.882563 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:13.899363 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:13.901463 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:13.912343 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:13.924237 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:13.947603 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.009222 27439 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "98bec1722172403aa21b3d060ed636d2"
dest_uuid: "aad738f0c10848789c2c811160990f8b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37998
I20240503 14:15:14.009397 27439 raft_consensus.cc:595] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:14.125030 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.125553 916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:14.135761 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.135761 27420 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:15:14.152943 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.153285 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.163826 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.172860 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.222553 1129 raft_consensus.cc:979] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: : Instructing follower 159f56e814134c3db8af6f1760255630 to start an election
I20240503 14:15:14.222795 1129 raft_consensus.cc:1067] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Signalling peer 159f56e814134c3db8af6f1760255630 to start an election
I20240503 14:15:14.223649 936 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "98bec1722172403aa21b3d060ed636d2"
dest_uuid: "159f56e814134c3db8af6f1760255630"
from {username='slave'} at 127.20.99.2:38261
I20240503 14:15:14.223958 936 raft_consensus.cc:482] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:14.224159 936 raft_consensus.cc:3037] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.227131 936 raft_consensus.cc:504] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:14.229851 27439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98bec1722172403aa21b3d060ed636d2" candidate_uuid: "159f56e814134c3db8af6f1760255630" candidate_term: 2 candidate_status { last_received { term: 1 index: 1479 } } ignore_live_leader: true dest_uuid: "aad738f0c10848789c2c811160990f8b"
I20240503 14:15:14.232983 936 leader_election.cc:288] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [CANDIDATE]: Term 2 election: Requested vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553), d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731)
I20240503 14:15:14.239797 27439 raft_consensus.cc:3032] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:14.239931 27439 raft_consensus.cc:729] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aad738f0c10848789c2c811160990f8b, State: Running, Role: LEADER
I20240503 14:15:14.240217 27439 consensus_queue.cc:260] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1479, Committed index: 1479, Last appended: 1.1479, Last appended by leader: 1479, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:14.240646 27439 raft_consensus.cc:3037] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.243886 27439 raft_consensus.cc:2454] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159f56e814134c3db8af6f1760255630 in term 2.
I20240503 14:15:14.244935 870 leader_election.cc:302] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159f56e814134c3db8af6f1760255630, aad738f0c10848789c2c811160990f8b; no voters:
I20240503 14:15:14.257972 1618 raft_consensus.cc:2781] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:14.261351 28734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98bec1722172403aa21b3d060ed636d2" candidate_uuid: "159f56e814134c3db8af6f1760255630" candidate_term: 2 candidate_status { last_received { term: 1 index: 1479 } } ignore_live_leader: true dest_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1"
I20240503 14:15:14.261575 28734 raft_consensus.cc:3037] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.263273 1618 raft_consensus.cc:686] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 2 LEADER]: Becoming Leader. State: Replica: 159f56e814134c3db8af6f1760255630, State: Running, Role: LEADER
I20240503 14:15:14.263836 1618 consensus_queue.cc:237] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1479, Committed index: 1479, Last appended: 1.1479, Last appended by leader: 1479, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:14.264535 28734 raft_consensus.cc:2454] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 159f56e814134c3db8af6f1760255630 in term 2.
I20240503 14:15:14.267959 27169 catalog_manager.cc:5484] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 reported cstate change: term changed from 1 to 2, leader changed from aad738f0c10848789c2c811160990f8b (127.20.99.2) to 159f56e814134c3db8af6f1760255630 (127.20.99.8). New cstate: current_term: 2 leader_uuid: "159f56e814134c3db8af6f1760255630" committed_config { opid_index: 1479 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:14.422499 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.433020 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.434262 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.440482 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.447270 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.464906 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.474148 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.493289 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.558177 27571 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd"
dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56974
I20240503 14:15:14.558403 27571 raft_consensus.cc:595] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:15:14.663367 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.665037 27552 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.666580 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.669324 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.701123 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:14.724093 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.726109 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:14.738938 28975 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:14.805706 28734 raft_consensus.cc:1261] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Refusing update from remote peer 159f56e814134c3db8af6f1760255630: Log matching property violated. Preceding OpId in replica: term: 1 index: 1479. Preceding OpId from leader: term: 2 index: 1480. (index mismatch)
I20240503 14:15:14.806496 1711 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1480, Last known committed idx: 1479, Time since last communication: 0.000s
I20240503 14:15:14.812413 27439 raft_consensus.cc:1261] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Refusing update from remote peer 159f56e814134c3db8af6f1760255630: Log matching property violated. Preceding OpId in replica: term: 1 index: 1479. Preceding OpId from leader: term: 2 index: 1480. (index mismatch)
I20240503 14:15:14.812980 1711 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1480, Last known committed idx: 1479, Time since last communication: 0.000s
I20240503 14:15:14.903733 1110 raft_consensus.cc:979] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: : Instructing follower dac9fd6f6a404989ae1a460ac3cde0fe to start an election
I20240503 14:15:14.903877 1110 raft_consensus.cc:1067] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Signalling peer dac9fd6f6a404989ae1a460ac3cde0fe to start an election
I20240503 14:15:14.904546 28995 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd"
dest_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe"
from {username='slave'} at 127.20.99.3:34287
I20240503 14:15:14.904768 28995 raft_consensus.cc:482] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:14.904889 28995 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:15:14.906821 28995 raft_consensus.cc:504] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:14.907616 28995 leader_election.cc:288] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 3 election: Requested vote from peers 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621), 159f56e814134c3db8af6f1760255630 (127.20.99.8:32857)
I20240503 14:15:14.908079 27571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" candidate_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 1428 } } ignore_live_leader: true dest_uuid: "2116de1ecc2f48daa3c8cca7edc82a95"
I20240503 14:15:14.908295 27571 raft_consensus.cc:3032] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:15:14.908391 27571 raft_consensus.cc:729] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2116de1ecc2f48daa3c8cca7edc82a95, State: Running, Role: LEADER
I20240503 14:15:14.908353 936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8102c1d08074ea7bb5be6c5645d1fcd" candidate_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 1428 } } ignore_live_leader: true dest_uuid: "159f56e814134c3db8af6f1760255630"
I20240503 14:15:14.908699 936 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:15:14.908677 27571 consensus_queue.cc:260] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1428, Committed index: 1428, Last appended: 2.1428, Last appended by leader: 1428, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:14.909078 27571 raft_consensus.cc:3037] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:15:14.911563 936 raft_consensus.cc:2454] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac9fd6f6a404989ae1a460ac3cde0fe in term 3.
I20240503 14:15:14.911890 27571 raft_consensus.cc:2454] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dac9fd6f6a404989ae1a460ac3cde0fe in term 3.
I20240503 14:15:14.912057 28931 leader_election.cc:302] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 159f56e814134c3db8af6f1760255630, dac9fd6f6a404989ae1a460ac3cde0fe; no voters:
I20240503 14:15:14.912360 1150 raft_consensus.cc:2781] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:15:14.912791 1150 raft_consensus.cc:686] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 3 LEADER]: Becoming Leader. State: Replica: dac9fd6f6a404989ae1a460ac3cde0fe, State: Running, Role: LEADER
I20240503 14:15:14.913059 1150 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1428, Committed index: 1428, Last appended: 2.1428, Last appended by leader: 1428, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:14.915607 27169 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: term changed from 2 to 3, leader changed from 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) to dac9fd6f6a404989ae1a460ac3cde0fe (127.20.99.7). New cstate: current_term: 3 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 1428 OBSOLETE_local: false peers { permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.108971 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:15.120563 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.126201 27552 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:15.129802 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.129963 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.132170 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.151634 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.257704 28975 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.338759 28995 consensus_queue.cc:237] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1542, Committed index: 1542, Last appended: 2.1542, Last appended by leader: 763, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1543 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:15.340113 936 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 3 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 1428. Preceding OpId from leader: term: 3 index: 1429. (index mismatch)
I20240503 14:15:15.340708 936 raft_consensus.cc:1261] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 1542. Preceding OpId from leader: term: 2 index: 1543. (index mismatch)
I20240503 14:15:15.340691 1150 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1429, Last known committed idx: 1428, Time since last communication: 0.000s
I20240503 14:15:15.341099 1150 consensus_queue.cc:1035] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1543, Last known committed idx: 1542, Time since last communication: 0.000s
I20240503 14:15:15.358402 1783 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe [term 2 LEADER]: Committing config change with OpId 2.1543: config changed from index 1542 to 1543, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 1543 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.367102 27571 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 3 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 1428. Preceding OpId from leader: term: 3 index: 1429. (index mismatch)
I20240503 14:15:15.367710 27162 catalog_manager.cc:5484] T a352cdeffa9a4b85badcefa545885bc3 P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 1542 to 1543, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 1543 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.367328 935 raft_consensus.cc:2932] T a352cdeffa9a4b85badcefa545885bc3 P 159f56e814134c3db8af6f1760255630 [term 2 FOLLOWER]: Committing config change with OpId 2.1543: config changed from index 1542 to 1543, VOTER 60da896b8369489994355cc0e9050514 (127.20.99.1) evicted. New config: { opid_index: 1543 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.368309 1786 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2116de1ecc2f48daa3c8cca7edc82a95" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1429, Last known committed idx: 1428, Time since last communication: 0.000s
I20240503 14:15:15.373694 27291 tablet_service.cc:1514] Processing DeleteTablet for tablet a352cdeffa9a4b85badcefa545885bc3 with delete_type TABLET_DATA_TOMBSTONED (TS 60da896b8369489994355cc0e9050514 not found in new config with opid_index 1543) from {username='slave'} at 127.0.0.1:60240
I20240503 14:15:15.382963 1788 tablet_replica.cc:330] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: stopping tablet replica
I20240503 14:15:15.383208 1788 raft_consensus.cc:2229] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:15.383476 1788 raft_consensus.cc:2258] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:15.384974 1788 ts_tablet_manager.cc:1875] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:15.391768 1788 ts_tablet_manager.cc:1888] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1542
I20240503 14:15:15.391918 1788 log.cc:1196] T a352cdeffa9a4b85badcefa545885bc3 P 60da896b8369489994355cc0e9050514: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/a352cdeffa9a4b85badcefa545885bc3
I20240503 14:15:15.392874 27154 catalog_manager.cc:4830] TS 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): tablet a352cdeffa9a4b85badcefa545885bc3 (table rebalance_test_table_1 [id=787e6cc6dc91431096701f08d93d48a2]) successfully deleted
I20240503 14:15:15.508244 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.509506 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:15.525236 27552 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:15.528836 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.548125 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.550045 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.552606 28975 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.552747 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.659472 935 consensus_queue.cc:237] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1480, Committed index: 1480, Last appended: 2.1480, Last appended by leader: 1479, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1481 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:15.661123 28734 raft_consensus.cc:1261] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Refusing update from remote peer 159f56e814134c3db8af6f1760255630: Log matching property violated. Preceding OpId in replica: term: 2 index: 1480. Preceding OpId from leader: term: 2 index: 1481. (index mismatch)
I20240503 14:15:15.661674 1711 consensus_queue.cc:1035] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1481, Last known committed idx: 1480, Time since last communication: 0.000s
I20240503 14:15:15.664274 1849 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 [term 2 LEADER]: Committing config change with OpId 2.1481: config changed from index 1479 to 1481, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1481 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.664866 28734 raft_consensus.cc:2932] T 98bec1722172403aa21b3d060ed636d2 P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Committing config change with OpId 2.1481: config changed from index 1479 to 1481, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New config: { opid_index: 1481 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.670359 27167 catalog_manager.cc:5484] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 reported cstate change: config changed from index 1479 to 1481, VOTER aad738f0c10848789c2c811160990f8b (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "159f56e814134c3db8af6f1760255630" committed_config { opid_index: 1481 OBSOLETE_local: false peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.675076 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet 98bec1722172403aa21b3d060ed636d2 with delete_type TABLET_DATA_TOMBSTONED (TS aad738f0c10848789c2c811160990f8b not found in new config with opid_index 1481) from {username='slave'} at 127.0.0.1:50958
I20240503 14:15:15.675724 1852 tablet_replica.cc:330] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: stopping tablet replica
I20240503 14:15:15.675895 1852 raft_consensus.cc:2229] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:15.676048 1852 raft_consensus.cc:2258] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:15.677662 1852 ts_tablet_manager.cc:1875] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:15.683368 1852 ts_tablet_manager.cc:1888] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1480
I20240503 14:15:15.683494 1852 log.cc:1196] T 98bec1722172403aa21b3d060ed636d2 P aad738f0c10848789c2c811160990f8b: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/98bec1722172403aa21b3d060ed636d2
I20240503 14:15:15.684248 27155 catalog_manager.cc:4830] TS aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): tablet 98bec1722172403aa21b3d060ed636d2 (table rebalance_test_table_3 [id=a1fa53f2cf2b43d8be80105923d28618]) successfully deleted
I20240503 14:15:15.786015 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:15.801589 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.802395 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.805699 27552 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:15.813310 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.821888 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.830021 28975 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:15.837844 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:15.949244 28995 consensus_queue.cc:237] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1429, Committed index: 1429, Last appended: 3.1429, Last appended by leader: 1428, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1430 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } }
I20240503 14:15:15.950830 935 raft_consensus.cc:1261] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 3 FOLLOWER]: Refusing update from remote peer dac9fd6f6a404989ae1a460ac3cde0fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 1429. Preceding OpId from leader: term: 3 index: 1430. (index mismatch)
I20240503 14:15:15.951453 1786 consensus_queue.cc:1035] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1430, Last known committed idx: 1429, Time since last communication: 0.000s
I20240503 14:15:15.953795 1903 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe [term 3 LEADER]: Committing config change with OpId 3.1430: config changed from index 1428 to 1430, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 1430 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.954330 935 raft_consensus.cc:2932] T e8102c1d08074ea7bb5be6c5645d1fcd P 159f56e814134c3db8af6f1760255630 [term 3 FOLLOWER]: Committing config change with OpId 3.1430: config changed from index 1428 to 1430, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New config: { opid_index: 1430 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.957962 27169 catalog_manager.cc:5484] T e8102c1d08074ea7bb5be6c5645d1fcd P dac9fd6f6a404989ae1a460ac3cde0fe reported cstate change: config changed from index 1428 to 1430, VOTER 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3) evicted. New cstate: current_term: 3 leader_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" committed_config { opid_index: 1430 OBSOLETE_local: false peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } } peers { permanent_uuid: "159f56e814134c3db8af6f1760255630" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 32857 } } }
I20240503 14:15:15.962059 27552 tablet_service.cc:1514] Processing DeleteTablet for tablet e8102c1d08074ea7bb5be6c5645d1fcd with delete_type TABLET_DATA_TOMBSTONED (TS 2116de1ecc2f48daa3c8cca7edc82a95 not found in new config with opid_index 1430) from {username='slave'} at 127.0.0.1:34100
I20240503 14:15:15.962647 1905 tablet_replica.cc:330] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: stopping tablet replica
I20240503 14:15:15.962881 1905 raft_consensus.cc:2229] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:15.963114 1905 raft_consensus.cc:2258] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:15.964905 1905 ts_tablet_manager.cc:1875] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:15.971071 1905 ts_tablet_manager.cc:1888] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1429
I20240503 14:15:15.971256 1905 log.cc:1196] T e8102c1d08074ea7bb5be6c5645d1fcd P 2116de1ecc2f48daa3c8cca7edc82a95: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:15:15.972098 27157 catalog_manager.cc:4830] TS 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): tablet e8102c1d08074ea7bb5be6c5645d1fcd (table rebalance_test_table_4 [id=5ef2a6e4b3704917a5e6e7baf23cb90b]) successfully deleted
I20240503 14:15:16.100656 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:16.107721 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.115101 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.118450 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.121701 27552 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:16.129832 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.150216 28975 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.226488 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.395012 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.405901 27552 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:16.415889 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.424633 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:16.425750 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.428474 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.441780 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.459772 28975 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.813457 28845 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:16.823962 916 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.843137 27291 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.850849 27552 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:15:16.875372 27420 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:15:16.886159 28715 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.891131 28584 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:15:16.908699 28975 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
16e0966be88a4fdfbedfd01d9ae02384 | 127.20.99.62:38501 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.99.20:46773 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
01c7323dc6f84fe0bc5eca4fcad8f078 | 127.20.99.6:45729 | HEALTHY | <none> | 2 | 0
159f56e814134c3db8af6f1760255630 | 127.20.99.8:32857 | HEALTHY | <none> | 1 | 0
2116de1ecc2f48daa3c8cca7edc82a95 | 127.20.99.3:43621 | HEALTHY | <none> | 2 | 0
60da896b8369489994355cc0e9050514 | 127.20.99.1:45565 | HEALTHY | <none> | 3 | 0
aad738f0c10848789c2c811160990f8b | 127.20.99.2:42553 | HEALTHY | <none> | 3 | 0
c57eaae6a75a43679120a10c6d3090c7 | 127.20.99.4:44243 | HEALTHY | <none> | 1 | 0
d7a1a0df5dd047ddbacafd0870ed01d1 | 127.20.99.5:37731 | HEALTHY | <none> | 1 | 0
dac9fd6f6a404989ae1a460ac3cde0fe | 127.20.99.7:45199 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.20.99.1 | experimental | 127.20.99.1:45565
local_ip_for_outbound_sockets | 127.20.99.2 | experimental | 127.20.99.2:42553
local_ip_for_outbound_sockets | 127.20.99.3 | experimental | 127.20.99.3:43621
local_ip_for_outbound_sockets | 127.20.99.4 | experimental | 127.20.99.4:44243
local_ip_for_outbound_sockets | 127.20.99.5 | experimental | 127.20.99.5:37731
local_ip_for_outbound_sockets | 127.20.99.6 | experimental | 127.20.99.6:45729
local_ip_for_outbound_sockets | 127.20.99.7 | experimental | 127.20.99.7:45199
local_ip_for_outbound_sockets | 127.20.99.8 | experimental | 127.20.99.8:32857
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.99.1:45565
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.99.2:42553
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.99.3:43621
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.99.4:44243
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.99.5:37731
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.99.6:45729
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.99.7:45199
server_dump_info_path | /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.20.99.8:32857
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.20.99.20:46773 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:15:16.968849 20876 log_verifier.cc:126] Checking tablet 244c146eb7624b3585068f9c2de0c4ed
I20240503 14:15:17.487898 20876 log_verifier.cc:177] Verified matching terms for 1514 ops in tablet 244c146eb7624b3585068f9c2de0c4ed
I20240503 14:15:17.489528 20876 log_verifier.cc:126] Checking tablet 2c65982ba3034daeacd3407a5b6877e1
I20240503 14:15:17.999810 20876 log_verifier.cc:177] Verified matching terms for 1469 ops in tablet 2c65982ba3034daeacd3407a5b6877e1
I20240503 14:15:18.001545 20876 log_verifier.cc:126] Checking tablet 4180d30f1d3449ff9c28c69599098ff4
I20240503 14:15:18.438349 20876 log_verifier.cc:177] Verified matching terms for 1409 ops in tablet 4180d30f1d3449ff9c28c69599098ff4
I20240503 14:15:18.439901 20876 log_verifier.cc:126] Checking tablet 44eb2dd4af5444f2a6a75b50bce0e03f
I20240503 14:15:18.449970 20876 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44eb2dd4af5444f2a6a75b50bce0e03f
I20240503 14:15:18.450064 20876 log_verifier.cc:126] Checking tablet 54fbf7658e87471895a21e486f59352a
I20240503 14:15:19.026456 20876 log_verifier.cc:177] Verified matching terms for 1842 ops in tablet 54fbf7658e87471895a21e486f59352a
I20240503 14:15:19.028558 20876 log_verifier.cc:126] Checking tablet 63dc99c47a8a4f379b3349837d4b7264
I20240503 14:15:19.279460 27488 maintenance_manager.cc:419] P aad738f0c10848789c2c811160990f8b: Scheduling FlushMRSOp(b44e3d7641cc42fdb789fbfd0405f072): perf score=0.033346
I20240503 14:15:19.400146 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling FlushMRSOp(54fbf7658e87471895a21e486f59352a): perf score=0.033374
I20240503 14:15:19.457501 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling FlushMRSOp(54fbf7658e87471895a21e486f59352a): perf score=0.033398
I20240503 14:15:19.529057 27379 maintenance_manager.cc:643] P aad738f0c10848789c2c811160990f8b: FlushMRSOp(b44e3d7641cc42fdb789fbfd0405f072) complete. Timing: real 0.249s user 0.160s sys 0.064s Metrics: {"bytes_written":54390,"cfile_init":1,"compiler_manager_pool.queue_time_us":375,"dirs.queue_time_us":6386,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":10511,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1217,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"rows_written":2376,"thread_start_us":40366,"threads_started":3,"wal-append.queue_time_us":33649}
I20240503 14:15:19.531057 27488 maintenance_manager.cc:419] P aad738f0c10848789c2c811160990f8b: Scheduling UndoDeltaBlockGCOp(b44e3d7641cc42fdb789fbfd0405f072): 24479 bytes on disk
I20240503 14:15:19.540053 27379 maintenance_manager.cc:643] P aad738f0c10848789c2c811160990f8b: UndoDeltaBlockGCOp(b44e3d7641cc42fdb789fbfd0405f072) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240503 14:15:19.799016 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: FlushMRSOp(54fbf7658e87471895a21e486f59352a) complete. Timing: real 0.398s user 0.179s sys 0.024s Metrics: {"bytes_written":57934,"cfile_init":1,"compiler_manager_pool.queue_time_us":384,"dirs.queue_time_us":10573,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":10623,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5998,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":2474,"thread_start_us":18675,"threads_started":3,"wal-append.queue_time_us":7086}
I20240503 14:15:19.801292 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling UndoDeltaBlockGCOp(54fbf7658e87471895a21e486f59352a): 25912 bytes on disk
I20240503 14:15:19.802425 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: UndoDeltaBlockGCOp(54fbf7658e87471895a21e486f59352a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 14:15:19.896122 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: FlushMRSOp(54fbf7658e87471895a21e486f59352a) complete. Timing: real 0.438s user 0.173s sys 0.033s Metrics: {"bytes_written":57934,"cfile_init":1,"compiler_manager_pool.queue_time_us":17464,"dirs.queue_time_us":3346,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":4552,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1235,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"rows_written":2474,"thread_start_us":35170,"threads_started":3,"wal-append.queue_time_us":9439}
I20240503 14:15:19.904428 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling UndoDeltaBlockGCOp(54fbf7658e87471895a21e486f59352a): 25912 bytes on disk
I20240503 14:15:19.905354 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: UndoDeltaBlockGCOp(54fbf7658e87471895a21e486f59352a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 14:15:19.994367 20876 log_verifier.cc:177] Verified matching terms for 1591 ops in tablet 63dc99c47a8a4f379b3349837d4b7264
I20240503 14:15:19.997862 20876 log_verifier.cc:126] Checking tablet 86890b3c77ba41d7b08579defb20aff1
I20240503 14:15:20.407253 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling FlushMRSOp(63dc99c47a8a4f379b3349837d4b7264): perf score=0.033367
I20240503 14:15:20.488363 20876 log_verifier.cc:177] Verified matching terms for 1249 ops in tablet 86890b3c77ba41d7b08579defb20aff1
I20240503 14:15:20.489817 20876 log_verifier.cc:126] Checking tablet 98bec1722172403aa21b3d060ed636d2
I20240503 14:15:20.498008 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: FlushMRSOp(63dc99c47a8a4f379b3349837d4b7264) complete. Timing: real 0.090s user 0.063s sys 0.025s Metrics: {"bytes_written":51612,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1530,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":2275,"thread_start_us":860,"threads_started":2,"wal-append.queue_time_us":741}
I20240503 14:15:20.498766 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling UndoDeltaBlockGCOp(63dc99c47a8a4f379b3349837d4b7264): 23153 bytes on disk
I20240503 14:15:20.499361 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: UndoDeltaBlockGCOp(63dc99c47a8a4f379b3349837d4b7264) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 14:15:20.543013 27488 maintenance_manager.cc:419] P aad738f0c10848789c2c811160990f8b: Scheduling FlushMRSOp(244c146eb7624b3585068f9c2de0c4ed): perf score=0.033401
I20240503 14:15:20.642544 27379 maintenance_manager.cc:643] P aad738f0c10848789c2c811160990f8b: FlushMRSOp(244c146eb7624b3585068f9c2de0c4ed) complete. Timing: real 0.099s user 0.072s sys 0.024s Metrics: {"bytes_written":49418,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1586,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":2143,"thread_start_us":501,"threads_started":2,"wal-append.queue_time_us":368}
I20240503 14:15:20.643633 27488 maintenance_manager.cc:419] P aad738f0c10848789c2c811160990f8b: Scheduling UndoDeltaBlockGCOp(244c146eb7624b3585068f9c2de0c4ed): 21878 bytes on disk
I20240503 14:15:20.644233 27379 maintenance_manager.cc:643] P aad738f0c10848789c2c811160990f8b: UndoDeltaBlockGCOp(244c146eb7624b3585068f9c2de0c4ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 14:15:20.985852 20876 log_verifier.cc:177] Verified matching terms for 1481 ops in tablet 98bec1722172403aa21b3d060ed636d2
I20240503 14:15:20.987707 20876 log_verifier.cc:126] Checking tablet a352cdeffa9a4b85badcefa545885bc3
I20240503 14:15:21.470683 20876 log_verifier.cc:177] Verified matching terms for 1543 ops in tablet a352cdeffa9a4b85badcefa545885bc3
I20240503 14:15:21.472671 20876 log_verifier.cc:126] Checking tablet ae02587deb4a450da3171f30a1501950
I20240503 14:15:21.895669 20876 log_verifier.cc:177] Verified matching terms for 1251 ops in tablet ae02587deb4a450da3171f30a1501950
I20240503 14:15:21.897050 20876 log_verifier.cc:126] Checking tablet b44e3d7641cc42fdb789fbfd0405f072
I20240503 14:15:22.147959 27488 maintenance_manager.cc:419] P aad738f0c10848789c2c811160990f8b: Scheduling FlushMRSOp(ae02587deb4a450da3171f30a1501950): perf score=0.033335
I20240503 14:15:22.253247 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling FlushMRSOp(86890b3c77ba41d7b08579defb20aff1): perf score=0.033362
I20240503 14:15:22.255256 27379 maintenance_manager.cc:643] P aad738f0c10848789c2c811160990f8b: FlushMRSOp(ae02587deb4a450da3171f30a1501950) complete. Timing: real 0.107s user 0.074s sys 0.032s Metrics: {"bytes_written":42813,"cfile_init":1,"dirs.queue_time_us":1372,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1491,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":750,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":1937,"thread_start_us":1515,"threads_started":2,"wal-append.queue_time_us":365}
I20240503 14:15:22.256021 27488 maintenance_manager.cc:419] P aad738f0c10848789c2c811160990f8b: Scheduling UndoDeltaBlockGCOp(ae02587deb4a450da3171f30a1501950): 19255 bytes on disk
I20240503 14:15:22.256469 27379 maintenance_manager.cc:643] P aad738f0c10848789c2c811160990f8b: UndoDeltaBlockGCOp(ae02587deb4a450da3171f30a1501950) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 14:15:22.308588 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling FlushMRSOp(ae02587deb4a450da3171f30a1501950): perf score=0.033376
I20240503 14:15:22.382457 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: FlushMRSOp(86890b3c77ba41d7b08579defb20aff1) complete. Timing: real 0.129s user 0.106s sys 0.019s Metrics: {"bytes_written":42642,"cfile_init":1,"dirs.queue_time_us":1721,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1578,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1108,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":1879,"thread_start_us":1894,"threads_started":2,"wal-append.queue_time_us":440}
I20240503 14:15:22.383533 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling UndoDeltaBlockGCOp(86890b3c77ba41d7b08579defb20aff1): 18875 bytes on disk
I20240503 14:15:22.384136 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: UndoDeltaBlockGCOp(86890b3c77ba41d7b08579defb20aff1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 14:15:22.428005 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: FlushMRSOp(ae02587deb4a450da3171f30a1501950) complete. Timing: real 0.119s user 0.079s sys 0.038s Metrics: {"bytes_written":42813,"cfile_init":1,"dirs.queue_time_us":366,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1547,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":1937,"thread_start_us":521,"threads_started":2,"wal-append.queue_time_us":351}
I20240503 14:15:22.428771 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling UndoDeltaBlockGCOp(ae02587deb4a450da3171f30a1501950): 19255 bytes on disk
I20240503 14:15:22.429316 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: UndoDeltaBlockGCOp(ae02587deb4a450da3171f30a1501950) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240503 14:15:22.469933 20876 log_verifier.cc:177] Verified matching terms for 1706 ops in tablet b44e3d7641cc42fdb789fbfd0405f072
I20240503 14:15:22.512270 20876 log_verifier.cc:126] Checking tablet c723f0969bd9472d833bdbb54ee67556
I20240503 14:15:22.951678 20876 log_verifier.cc:177] Verified matching terms for 1476 ops in tablet c723f0969bd9472d833bdbb54ee67556
I20240503 14:15:22.953295 20876 log_verifier.cc:126] Checking tablet dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:15:23.387097 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling FlushMRSOp(f576c9fe2ddf4494b51d41d64d18a1eb): perf score=0.033385
I20240503 14:15:23.424747 20876 log_verifier.cc:177] Verified matching terms for 1275 ops in tablet dd18d7c1f6f64ef08d227e039e369cbc
I20240503 14:15:23.426213 20876 log_verifier.cc:126] Checking tablet e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:15:23.431893 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling FlushMRSOp(c723f0969bd9472d833bdbb54ee67556): perf score=0.033395
I20240503 14:15:23.501497 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: FlushMRSOp(f576c9fe2ddf4494b51d41d64d18a1eb) complete. Timing: real 0.114s user 0.079s sys 0.032s Metrics: {"bytes_written":48326,"cfile_init":1,"dirs.queue_time_us":402,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":3995,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1104,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":2146,"thread_start_us":488,"threads_started":2,"wal-append.queue_time_us":372}
I20240503 14:15:23.502424 27618 maintenance_manager.cc:419] P 2116de1ecc2f48daa3c8cca7edc82a95: Scheduling UndoDeltaBlockGCOp(f576c9fe2ddf4494b51d41d64d18a1eb): 21674 bytes on disk
I20240503 14:15:23.503079 27509 maintenance_manager.cc:643] P 2116de1ecc2f48daa3c8cca7edc82a95: UndoDeltaBlockGCOp(f576c9fe2ddf4494b51d41d64d18a1eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.521978 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: FlushMRSOp(c723f0969bd9472d833bdbb54ee67556) complete. Timing: real 0.090s user 0.066s sys 0.020s Metrics: {"bytes_written":48504,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":2147,"thread_start_us":498,"threads_started":2,"wal-append.queue_time_us":346}
I20240503 14:15:23.523633 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling UndoDeltaBlockGCOp(c723f0969bd9472d833bdbb54ee67556): 21780 bytes on disk
I20240503 14:15:23.524356 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: UndoDeltaBlockGCOp(c723f0969bd9472d833bdbb54ee67556) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.889073 20876 log_verifier.cc:177] Verified matching terms for 1430 ops in tablet e8102c1d08074ea7bb5be6c5645d1fcd
I20240503 14:15:23.890653 20876 log_verifier.cc:126] Checking tablet f3923478811a4a0f82ec9f013ef7735d
I20240503 14:15:24.277804 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling FlushMRSOp(4180d30f1d3449ff9c28c69599098ff4): perf score=0.033341
I20240503 14:15:24.397001 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: FlushMRSOp(4180d30f1d3449ff9c28c69599098ff4) complete. Timing: real 0.119s user 0.101s sys 0.016s Metrics: {"bytes_written":46781,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1689,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1142,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"rows_written":2079,"thread_start_us":491,"threads_started":2,"wal-append.queue_time_us":365}
I20240503 14:15:24.397882 27357 maintenance_manager.cc:419] P 60da896b8369489994355cc0e9050514: Scheduling UndoDeltaBlockGCOp(4180d30f1d3449ff9c28c69599098ff4): 20965 bytes on disk
I20240503 14:15:24.398566 27248 maintenance_manager.cc:643] P 60da896b8369489994355cc0e9050514: UndoDeltaBlockGCOp(4180d30f1d3449ff9c28c69599098ff4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 14:15:24.540518 20876 log_verifier.cc:177] Verified matching terms for 1814 ops in tablet f3923478811a4a0f82ec9f013ef7735d
I20240503 14:15:24.542435 20876 log_verifier.cc:126] Checking tablet f576c9fe2ddf4494b51d41d64d18a1eb
I20240503 14:15:24.972283 20876 log_verifier.cc:177] Verified matching terms for 1463 ops in tablet f576c9fe2ddf4494b51d41d64d18a1eb
I20240503 14:15:24.976712 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27229
I20240503 14:15:24.997972 27350 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:25.335592 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27229
I20240503 14:15:25.392243 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27360
I20240503 14:15:25.397607 27482 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:25.727123 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27360
I20240503 14:15:25.779278 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27490
I20240503 14:15:25.784970 27611 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:15:25.833245 28929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.2:42553: connect: Connection refused (error 111)
W20240503 14:15:25.834645 28929 consensus_peers.cc:485] T b44e3d7641cc42fdb789fbfd0405f072 P dac9fd6f6a404989ae1a460ac3cde0fe -> Peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): Couldn't send request to peer aad738f0c10848789c2c811160990f8b. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.2:42553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:26.104262 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27490
W20240503 14:15:26.162102 28538 connection.cc:535] server connection from 127.20.99.3:44835 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:26.162746 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 28522
W20240503 14:15:26.165203 28801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.3:43621: connect: Connection refused (error 111)
W20240503 14:15:26.166489 28801 consensus_peers.cc:485] T f576c9fe2ddf4494b51d41d64d18a1eb P 01c7323dc6f84fe0bc5eca4fcad8f078 -> Peer 2116de1ecc2f48daa3c8cca7edc82a95 (127.20.99.3:43621): Couldn't send request to peer 2116de1ecc2f48daa3c8cca7edc82a95. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.3:43621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:26.168985 28644 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:26.358508 1960 raft_consensus.cc:482] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60da896b8369489994355cc0e9050514)
I20240503 14:15:26.358726 1960 raft_consensus.cc:504] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 839 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "01c7323dc6f84fe0bc5eca4fcad8f078" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 45729 } }
I20240503 14:15:26.359479 1960 leader_election.cc:288] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60da896b8369489994355cc0e9050514 (127.20.99.1:45565)
W20240503 14:15:26.360903 28798 leader_election.cc:334] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 60da896b8369489994355cc0e9050514 (127.20.99.1:45565): Network error: Client connection negotiation failed: client connection to 127.20.99.1:45565: connect: Connection refused (error 111)
I20240503 14:15:26.361083 28798 leader_election.cc:302] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01c7323dc6f84fe0bc5eca4fcad8f078; no voters: 60da896b8369489994355cc0e9050514
I20240503 14:15:26.361352 1960 raft_consensus.cc:2726] T 4180d30f1d3449ff9c28c69599098ff4 P 01c7323dc6f84fe0bc5eca4fcad8f078 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:15:26.454429 2091 raft_consensus.cc:482] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60da896b8369489994355cc0e9050514)
I20240503 14:15:26.454651 2091 raft_consensus.cc:504] T c723f0969bd9472d833bdbb54ee67556 P c57eaae6a75a43679120a10c6d3090c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "60da896b8369489994355cc0e9050514" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 45565 } } peers { permanent_uuid: "c57eaae6a75a43679120a10c6d3090c7" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 44243 } }
I20240503 14:15:26.455276 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 28522
W20240503 14:15:26.504415 27157 connection.cc:535] server connection from 127.20.99.4:39679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:26.504762 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 28653
I20240503 14:15:26.510617 28774 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:26.777740 2094 raft_consensus.cc:482] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader aad738f0c10848789c2c811160990f8b)
I20240503 14:15:26.777958 2094 raft_consensus.cc:504] T 244c146eb7624b3585068f9c2de0c4ed P d7a1a0df5dd047ddbacafd0870ed01d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 621 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "d7a1a0df5dd047ddbacafd0870ed01d1" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 37731 } }
I20240503 14:15:26.778321 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 28653
W20240503 14:15:26.824824 869 connection.cc:535] client connection to 127.20.99.5:37731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:26.824824 27157 connection.cc:535] server connection from 127.20.99.5:47621 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:26.825094 869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:26.825204 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 28783
W20240503 14:15:26.827252 869 consensus_peers.cc:485] T 98bec1722172403aa21b3d060ed636d2 P 159f56e814134c3db8af6f1760255630 -> Peer d7a1a0df5dd047ddbacafd0870ed01d1 (127.20.99.5:37731): Couldn't send request to peer d7a1a0df5dd047ddbacafd0870ed01d1. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:37731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:26.831091 28905 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:26.892284 2022 raft_consensus.cc:482] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aad738f0c10848789c2c811160990f8b)
I20240503 14:15:26.892576 2022 raft_consensus.cc:504] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aad738f0c10848789c2c811160990f8b" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 42553 } } peers { permanent_uuid: "dac9fd6f6a404989ae1a460ac3cde0fe" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 45199 } }
I20240503 14:15:26.893196 2022 leader_election.cc:288] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad738f0c10848789c2c811160990f8b (127.20.99.2:42553)
W20240503 14:15:26.894604 28929 leader_election.cc:334] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aad738f0c10848789c2c811160990f8b (127.20.99.2:42553): Network error: Client connection negotiation failed: client connection to 127.20.99.2:42553: connect: Connection refused (error 111)
I20240503 14:15:26.894752 28929 leader_election.cc:302] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: dac9fd6f6a404989ae1a460ac3cde0fe; no voters: aad738f0c10848789c2c811160990f8b
I20240503 14:15:26.895007 2022 raft_consensus.cc:2726] T 44eb2dd4af5444f2a6a75b50bce0e03f P dac9fd6f6a404989ae1a460ac3cde0fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:27.101352 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 28783
I20240503 14:15:27.149667 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 28913
I20240503 14:15:27.155165 29035 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:27.424192 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 28913
I20240503 14:15:27.465133 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 853
I20240503 14:15:27.470894 975 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:27.679590 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 853
I20240503 14:15:27.712656 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 27139
I20240503 14:15:27.713668 27200 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:27.964525 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 27139
2024-05-03T14:15:28Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (136958 ms)
[----------] 1 test from RebalanceParamTest (136958 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240503 14:15:28.092653 20876 test_util.cc:268] Using random seed: 1510001105
I20240503 14:15:28.095688 20876 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:15:28.095841 20876 ts_itest-base.cc:113] --------------
I20240503 14:15:28.095906 20876 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:15:28.096005 20876 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:15:28.096107 20876 ts_itest-base.cc:116] --------------
2024-05-03T14:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:15:28Z Disabled control of system clock
I20240503 14:15:28.166757 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:33535
--webserver_interface=127.20.99.62
--webserver_port=0
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.99.62:33535
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 14:15:28.518229 2129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:28.518570 2129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:28.518699 2129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:28.526341 2129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:28.526479 2129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:15:28.526546 2129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:28.526630 2129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:15:28.526691 2129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:15:28.538059 2129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.99.62:33535
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.99.62:33535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.99.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:28.539790 2129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:28.542150 2129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:28.553220 2134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:28.554205 2135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:28.555600 2137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:28.555738 2129 server_base.cc:1034] running on GCE node
I20240503 14:15:28.556490 2129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:28.557866 2129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:28.559073 2129 hybrid_clock.cc:648] HybridClock initialized: now 1714745728559028 us; error 45 us; skew 500 ppm
I20240503 14:15:28.559435 2129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:28.567634 2129 webserver.cc:460] Webserver started at http://127.20.99.62:33471/ using document root <none> and password file <none>
I20240503 14:15:28.568387 2129 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:28.568507 2129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:28.568832 2129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:28.571516 2129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "13adeda3db494b478178ddd5ee8fdb98"
format_stamp: "Formatted at 2024-05-03 14:15:28 on dist-test-slave-p0gh"
I20240503 14:15:28.572274 2129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "13adeda3db494b478178ddd5ee8fdb98"
format_stamp: "Formatted at 2024-05-03 14:15:28 on dist-test-slave-p0gh"
I20240503 14:15:28.577452 2129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:15:28.580770 2143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:28.582002 2129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:15:28.582161 2129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
uuid: "13adeda3db494b478178ddd5ee8fdb98"
format_stamp: "Formatted at 2024-05-03 14:15:28 on dist-test-slave-p0gh"
I20240503 14:15:28.582348 2129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:28.595448 2129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:28.596259 2129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:28.596521 2129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:28.614490 2129 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.62:33535
I20240503 14:15:28.614507 2194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.62:33535 every 8 connection(s)
I20240503 14:15:28.615909 2129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:15:28.619674 2195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:28.625236 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2129
I20240503 14:15:28.625471 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:15:28.630277 2195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98: Bootstrap starting.
I20240503 14:15:28.633679 2195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:28.634944 2195 log.cc:824] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:28.638222 2195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98: No bootstrap required, opened a new log
I20240503 14:15:28.644591 2195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13adeda3db494b478178ddd5ee8fdb98" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 33535 } }
I20240503 14:15:28.644930 2195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:28.645031 2195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13adeda3db494b478178ddd5ee8fdb98, State: Initialized, Role: FOLLOWER
I20240503 14:15:28.645610 2195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13adeda3db494b478178ddd5ee8fdb98" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 33535 } }
I20240503 14:15:28.645802 2195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:28.645925 2195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:28.646078 2195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:28.648257 2195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13adeda3db494b478178ddd5ee8fdb98" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 33535 } }
I20240503 14:15:28.648741 2195 leader_election.cc:302] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13adeda3db494b478178ddd5ee8fdb98; no voters:
I20240503 14:15:28.649302 2195 leader_election.cc:288] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:28.649434 2200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:28.650171 2200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [term 1 LEADER]: Becoming Leader. State: Replica: 13adeda3db494b478178ddd5ee8fdb98, State: Running, Role: LEADER
I20240503 14:15:28.650756 2200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13adeda3db494b478178ddd5ee8fdb98" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 33535 } }
I20240503 14:15:28.651463 2195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:15:28.653800 2201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13adeda3db494b478178ddd5ee8fdb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13adeda3db494b478178ddd5ee8fdb98" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 33535 } } }
I20240503 14:15:28.653805 2202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13adeda3db494b478178ddd5ee8fdb98. Latest consensus state: current_term: 1 leader_uuid: "13adeda3db494b478178ddd5ee8fdb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13adeda3db494b478178ddd5ee8fdb98" member_type: VOTER last_known_addr { host: "127.20.99.62" port: 33535 } } }
I20240503 14:15:28.654217 2201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:28.654256 2202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98 [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:28.657773 2206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:15:28.663125 2206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:15:28.672609 2206 catalog_manager.cc:1347] Generated new cluster ID: 1b90986f8bad4655a1043df528cf1cf9
I20240503 14:15:28.672748 2206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:15:28.709980 2206 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:15:28.711840 2206 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:15:28.719651 2206 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 13adeda3db494b478178ddd5ee8fdb98: Generated new TSK 0
I20240503 14:15:28.720598 2206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:15:28.736222 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.1:0
--local_ip_for_outbound_sockets=127.20.99.1
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:15:29.108319 2219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:29.108637 2219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:29.108727 2219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:29.116231 2219 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:29.116360 2219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:29.116530 2219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.1
I20240503 14:15:29.125792 2219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.99.1
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.1
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:29.127210 2219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:29.128942 2219 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:29.141150 2225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:29.141999 2224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:29.142670 2227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:29.143795 2219 server_base.cc:1034] running on GCE node
I20240503 14:15:29.144399 2219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:29.145309 2219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:29.146533 2219 hybrid_clock.cc:648] HybridClock initialized: now 1714745729146472 us; error 38 us; skew 500 ppm
I20240503 14:15:29.146992 2219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:29.149669 2219 webserver.cc:460] Webserver started at http://127.20.99.1:41475/ using document root <none> and password file <none>
I20240503 14:15:29.150406 2219 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:29.150522 2219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:29.150857 2219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:29.153632 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-p0gh"
I20240503 14:15:29.154415 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-p0gh"
I20240503 14:15:29.159874 2219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 14:15:29.163252 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.164590 2219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:29.164749 2219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-p0gh"
I20240503 14:15:29.164958 2219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:29.186244 2219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:29.187106 2219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:29.187403 2219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:29.188356 2219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:29.189862 2219 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:29.189949 2219 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.190064 2219 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:29.190150 2219 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.221498 2219 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.1:33597
I20240503 14:15:29.221557 2345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.1:33597 every 8 connection(s)
I20240503 14:15:29.222915 2219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:15:29.224403 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2219
I20240503 14:15:29.224584 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:15:29.228385 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.2:0
--local_ip_for_outbound_sockets=127.20.99.2
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:29.237577 2346 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:29.237876 2346 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:29.238586 2346 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:29.240772 2160 ts_manager.cc:194] Registered new tserver with Master: 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:29.243000 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.1:45269
W20240503 14:15:29.581380 2350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:29.581678 2350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:29.581785 2350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:29.589403 2350 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:29.589526 2350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:29.589649 2350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.2
I20240503 14:15:29.598659 2350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.99.2
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.2
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:29.599982 2350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:29.601634 2350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:29.613466 2355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:29.614449 2356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:29.615154 2358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:29.615958 2350 server_base.cc:1034] running on GCE node
I20240503 14:15:29.616516 2350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:29.617465 2350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:29.618669 2350 hybrid_clock.cc:648] HybridClock initialized: now 1714745729618609 us; error 57 us; skew 500 ppm
I20240503 14:15:29.618979 2350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:29.621987 2350 webserver.cc:460] Webserver started at http://127.20.99.2:41905/ using document root <none> and password file <none>
I20240503 14:15:29.622797 2350 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:29.622921 2350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:29.623278 2350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:29.625878 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "685c6181561b4056abdecfd6d8bdb3b2"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-p0gh"
I20240503 14:15:29.626649 2350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "685c6181561b4056abdecfd6d8bdb3b2"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-p0gh"
I20240503 14:15:29.631794 2350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:15:29.634954 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.636339 2350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:29.636508 2350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "685c6181561b4056abdecfd6d8bdb3b2"
format_stamp: "Formatted at 2024-05-03 14:15:29 on dist-test-slave-p0gh"
I20240503 14:15:29.636700 2350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:29.661317 2350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:29.662297 2350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:29.662585 2350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:29.663591 2350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:29.665125 2350 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:29.665226 2350 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.665338 2350 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:29.665407 2350 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:29.695084 2350 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.2:43457
I20240503 14:15:29.695132 2476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.2:43457 every 8 connection(s)
I20240503 14:15:29.696465 2350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:15:29.704361 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2350
I20240503 14:15:29.704530 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:15:29.707834 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.3:0
--local_ip_for_outbound_sockets=127.20.99.3
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:29.709307 2477 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:29.709617 2477 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:29.710345 2477 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:29.711700 2160 ts_manager.cc:194] Registered new tserver with Master: 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:29.712455 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.2:60219
W20240503 14:15:30.053133 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:30.053432 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:30.053529 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:30.060961 2480 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:30.061089 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:30.061206 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.3
I20240503 14:15:30.070454 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.99.3
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.3
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:30.071810 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:30.073465 2480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:30.085443 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.086448 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.087309 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:30.088124 2480 server_base.cc:1034] running on GCE node
I20240503 14:15:30.088763 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:30.089753 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:30.090991 2480 hybrid_clock.cc:648] HybridClock initialized: now 1714745730090941 us; error 45 us; skew 500 ppm
I20240503 14:15:30.091363 2480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:30.093822 2480 webserver.cc:460] Webserver started at http://127.20.99.3:39335/ using document root <none> and password file <none>
I20240503 14:15:30.094594 2480 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:30.094714 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:30.095028 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:30.098062 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b35c386497d046de9be5007a31ab7d10"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-p0gh"
I20240503 14:15:30.099114 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b35c386497d046de9be5007a31ab7d10"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-p0gh"
I20240503 14:15:30.104714 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240503 14:15:30.108117 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.109591 2480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:15:30.109789 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b35c386497d046de9be5007a31ab7d10"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-p0gh"
I20240503 14:15:30.109998 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:30.143002 2480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:30.143898 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:30.144155 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:30.145126 2480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:30.146600 2480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:30.146693 2480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.146800 2480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:30.146868 2480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.178349 2480 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.3:45269
I20240503 14:15:30.178409 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.3:45269 every 8 connection(s)
I20240503 14:15:30.179709 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:30.187958 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2480
I20240503 14:15:30.188138 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:15:30.191550 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.4:0
--local_ip_for_outbound_sockets=127.20.99.4
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:30.192348 2607 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:30.192632 2607 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:30.193395 2607 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:30.194839 2160 ts_manager.cc:194] Registered new tserver with Master: b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:30.195959 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.3:39699
I20240503 14:15:30.245903 2346 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
W20240503 14:15:30.536303 2610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:30.536576 2610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:30.536661 2610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:30.544194 2610 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:30.544348 2610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:30.544466 2610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.4
I20240503 14:15:30.554610 2610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.99.4
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.4
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:30.555945 2610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:30.557576 2610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:30.569869 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.570514 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:30.571344 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:30.572067 2610 server_base.cc:1034] running on GCE node
I20240503 14:15:30.572654 2610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:30.573583 2610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:30.574795 2610 hybrid_clock.cc:648] HybridClock initialized: now 1714745730574741 us; error 50 us; skew 500 ppm
I20240503 14:15:30.575114 2610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:30.577507 2610 webserver.cc:460] Webserver started at http://127.20.99.4:35925/ using document root <none> and password file <none>
I20240503 14:15:30.578262 2610 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:30.578379 2610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:30.578716 2610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:30.581336 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c37183c2965f46a28847c1262eb97829"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-p0gh"
I20240503 14:15:30.582121 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c37183c2965f46a28847c1262eb97829"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-p0gh"
I20240503 14:15:30.587402 2610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240503 14:15:30.590775 2624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.592319 2610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:15:30.592480 2610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c37183c2965f46a28847c1262eb97829"
format_stamp: "Formatted at 2024-05-03 14:15:30 on dist-test-slave-p0gh"
I20240503 14:15:30.592686 2610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:30.628870 2610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:30.629751 2610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:30.630031 2610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:30.630998 2610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:30.632521 2610 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:30.632618 2610 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.632730 2610 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:30.632798 2610 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:30.664780 2610 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.4:39091
I20240503 14:15:30.664852 2736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.4:39091 every 8 connection(s)
I20240503 14:15:30.666121 2610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:30.671311 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2610
I20240503 14:15:30.671497 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:15:30.674405 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:0
--local_ip_for_outbound_sockets=127.20.99.5
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:30.678838 2737 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:30.679250 2737 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:30.680203 2737 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:30.681763 2160 ts_manager.cc:194] Registered new tserver with Master: c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:30.682572 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.4:56109
I20240503 14:15:30.714912 2477 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
W20240503 14:15:31.031138 2740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:31.031596 2740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:31.031738 2740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:31.039273 2740 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:31.039430 2740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:31.039600 2740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:15:31.048803 2740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:31.050210 2740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:31.052196 2740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:31.064736 2745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.065730 2748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.066727 2746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:31.067518 2740 server_base.cc:1034] running on GCE node
I20240503 14:15:31.068161 2740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:31.069200 2740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:31.070403 2740 hybrid_clock.cc:648] HybridClock initialized: now 1714745731070342 us; error 64 us; skew 500 ppm
I20240503 14:15:31.070716 2740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:31.072953 2740 webserver.cc:460] Webserver started at http://127.20.99.5:41715/ using document root <none> and password file <none>
I20240503 14:15:31.073714 2740 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:31.073838 2740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:31.074165 2740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:31.076885 2740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7146de39e348442da4a363e95fbe0789"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:31.077669 2740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7146de39e348442da4a363e95fbe0789"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:31.083340 2740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 14:15:31.086941 2754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.088482 2740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:31.088665 2740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7146de39e348442da4a363e95fbe0789"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:31.088892 2740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:31.106981 2740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:31.107903 2740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:31.108192 2740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:31.109191 2740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:31.110699 2740 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:31.110803 2740 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.110919 2740 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:31.110996 2740 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.144459 2740 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:41737
I20240503 14:15:31.144549 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:41737 every 8 connection(s)
I20240503 14:15:31.145784 2740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:15:31.147639 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2740
I20240503 14:15:31.147820 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:15:31.151502 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.6:0
--local_ip_for_outbound_sockets=127.20.99.6
--webserver_interface=127.20.99.6
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:31.160106 2867 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:31.160408 2867 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:31.161144 2867 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:31.162649 2160 ts_manager.cc:194] Registered new tserver with Master: 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:15:31.163573 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:57499
I20240503 14:15:31.198580 2607 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
W20240503 14:15:31.508783 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:31.509084 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:31.509171 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:31.516884 2870 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:31.517016 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:31.517140 2870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.6
I20240503 14:15:31.526331 2870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.99.6
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.6
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:31.527776 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:31.529479 2870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:31.541918 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.542779 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:31.543598 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:31.544490 2870 server_base.cc:1034] running on GCE node
I20240503 14:15:31.545104 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:31.546080 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:31.547281 2870 hybrid_clock.cc:648] HybridClock initialized: now 1714745731547225 us; error 51 us; skew 500 ppm
I20240503 14:15:31.547595 2870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:31.550031 2870 webserver.cc:460] Webserver started at http://127.20.99.6:36511/ using document root <none> and password file <none>
I20240503 14:15:31.551030 2870 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:31.551134 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:31.551524 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:31.554636 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "285488446ac445c5a5dfe654bfaf9835"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:31.555505 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "285488446ac445c5a5dfe654bfaf9835"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:31.560545 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:15:31.563836 2884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.565107 2870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:31.565258 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "285488446ac445c5a5dfe654bfaf9835"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:31.565439 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:31.580539 2870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:31.581378 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:31.581668 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:31.582579 2870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:31.584089 2870 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:31.584193 2870 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.584331 2870 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:31.584404 2870 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:31.619539 2870 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.6:42763
I20240503 14:15:31.619573 2996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.6:42763 every 8 connection(s)
I20240503 14:15:31.620944 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:15:31.623132 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 2870
I20240503 14:15:31.623339 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:15:31.626796 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.7:0
--local_ip_for_outbound_sockets=127.20.99.7
--webserver_interface=127.20.99.7
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:31.635111 2997 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:31.635488 2997 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:31.636442 2997 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:31.637981 2160 ts_manager.cc:194] Registered new tserver with Master: 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:15:31.638765 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.6:46751
I20240503 14:15:31.685843 2737 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
W20240503 14:15:31.985347 3000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:31.985703 3000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:31.985843 3000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:31.993549 3000 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:31.993703 3000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:31.993873 3000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.7
I20240503 14:15:32.003031 3000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.99.7
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.7
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:32.004484 3000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:32.006245 3000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:32.019269 3006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.019697 3005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.020645 3008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:32.021656 3000 server_base.cc:1034] running on GCE node
I20240503 14:15:32.022296 3000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:32.023363 3000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:32.024569 3000 hybrid_clock.cc:648] HybridClock initialized: now 1714745732024519 us; error 48 us; skew 500 ppm
I20240503 14:15:32.024900 3000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:32.027292 3000 webserver.cc:460] Webserver started at http://127.20.99.7:39841/ using document root <none> and password file <none>
I20240503 14:15:32.028039 3000 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:32.028142 3000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:32.028453 3000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:32.031271 3000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d4cbc83202c8473d8c897582d30e5989"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:32.032075 3000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d4cbc83202c8473d8c897582d30e5989"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:32.037401 3000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:15:32.041008 3014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.042557 3000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:32.042721 3000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d4cbc83202c8473d8c897582d30e5989"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:32.042918 3000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:32.061231 3000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:32.062157 3000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:32.062428 3000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:32.063439 3000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:32.065063 3000 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:32.065150 3000 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.065253 3000 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:32.065325 3000 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.097301 3000 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.7:44387
I20240503 14:15:32.097343 3126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.7:44387 every 8 connection(s)
I20240503 14:15:32.098762 3000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:15:32.101037 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 3000
I20240503 14:15:32.101209 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:15:32.104797 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.8:0
--local_ip_for_outbound_sockets=127.20.99.8
--webserver_interface=127.20.99.8
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:32.112644 3127 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:32.113015 3127 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:32.114017 3127 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:32.115674 2160 ts_manager.cc:194] Registered new tserver with Master: d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:15:32.116830 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.7:55397
I20240503 14:15:32.166262 2867 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
W20240503 14:15:32.461730 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:32.462029 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:32.462114 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:32.469872 3130 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:32.469998 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:32.470120 3130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.8
I20240503 14:15:32.479243 3130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.99.8
--webserver_port=0
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.8
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:32.480546 3130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:32.482246 3130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:32.494177 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.495028 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:32.495865 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:32.496423 3130 server_base.cc:1034] running on GCE node
I20240503 14:15:32.497109 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:32.498147 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:32.499353 3130 hybrid_clock.cc:648] HybridClock initialized: now 1714745732499310 us; error 46 us; skew 500 ppm
I20240503 14:15:32.499686 3130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:32.501981 3130 webserver.cc:460] Webserver started at http://127.20.99.8:35637/ using document root <none> and password file <none>
I20240503 14:15:32.502727 3130 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:32.502840 3130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:32.503156 3130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:32.505828 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:32.506592 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:32.511888 3130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240503 14:15:32.515324 3144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.516682 3130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:32.516849 3130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:32.517048 3130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:32.553476 3130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:32.554596 3130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:32.554909 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:32.556172 3130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:32.557853 3130 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:32.557963 3130 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.558073 3130 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:32.558143 3130 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:32.590039 3130 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.8:45929
I20240503 14:15:32.590096 3256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.8:45929 every 8 connection(s)
I20240503 14:15:32.591379 3130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:15:32.598853 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 3130
I20240503 14:15:32.599028 20876 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:15:32.604054 3257 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:32.604363 3257 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:32.605098 3257 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:32.606350 2160 ts_manager.cc:194] Registered new tserver with Master: c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:15:32.607100 2160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.8:55957
I20240503 14:15:32.615442 20876 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:15:32.632759 20876 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:15:32.632891 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 2740
I20240503 14:15:32.640664 2862 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:32.641911 2997 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
I20240503 14:15:32.722342 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 2740
I20240503 14:15:32.739269 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 2870
I20240503 14:15:32.747454 2992 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:32.830663 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 2870
I20240503 14:15:32.847102 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 3000
I20240503 14:15:32.854976 3122 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:32.935925 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 3000
I20240503 14:15:32.952593 20876 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu pid 3130
I20240503 14:15:32.961099 3252 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:33.042760 20876 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu with pid 3130
I20240503 14:15:36.811527 2160 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:36.849393 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 6c94404d8d4f45d59958ae6feb36fc24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.850956 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c94404d8d4f45d59958ae6feb36fc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.851649 2411 tablet_service.cc:1467] Processing CreateTablet for tablet fe0eebf47c494dad8edf06197cdf15bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.852164 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 6c94404d8d4f45d59958ae6feb36fc24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.849633 2279 tablet_service.cc:1467] Processing CreateTablet for tablet fe0eebf47c494dad8edf06197cdf15bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.852881 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0eebf47c494dad8edf06197cdf15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.853068 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0eebf47c494dad8edf06197cdf15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.853502 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c94404d8d4f45d59958ae6feb36fc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.849335 2281 tablet_service.cc:1467] Processing CreateTablet for tablet dd294a04366145378e3f5e3d306152e3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.854044 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd294a04366145378e3f5e3d306152e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.854560 2672 tablet_service.cc:1467] Processing CreateTablet for tablet dd294a04366145378e3f5e3d306152e3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.855930 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd294a04366145378e3f5e3d306152e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.861040 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 6c94404d8d4f45d59958ae6feb36fc24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.861655 2671 tablet_service.cc:1467] Processing CreateTablet for tablet fe0eebf47c494dad8edf06197cdf15bc (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.862182 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0eebf47c494dad8edf06197cdf15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.862000 2542 tablet_service.cc:1467] Processing CreateTablet for tablet dd294a04366145378e3f5e3d306152e3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:36.864720 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c94404d8d4f45d59958ae6feb36fc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.867352 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd294a04366145378e3f5e3d306152e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.874118 3286 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:36.878756 3286 tablet_bootstrap.cc:654] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.881031 3286 log.cc:824] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:36.890434 3290 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:36.894500 3288 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:36.894490 3287 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:36.897923 3290 tablet_bootstrap.cc:654] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.897984 3287 tablet_bootstrap.cc:654] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.897984 3288 tablet_bootstrap.cc:654] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.899305 3288 log.cc:824] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:36.899300 3287 log.cc:824] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:36.899641 3290 log.cc:824] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:36.902913 3288 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:36.902967 3286 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:36.903120 3290 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:36.903278 3288 ts_tablet_manager.cc:1367] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:15:36.903393 3290 ts_tablet_manager.cc:1367] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240503 14:15:36.903414 3286 ts_tablet_manager.cc:1367] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.006s
I20240503 14:15:36.906164 3287 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:36.906461 3287 ts_tablet_manager.cc:1367] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 14:15:36.912290 3290 raft_consensus.cc:348] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.912292 3286 raft_consensus.cc:348] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.912281 3288 raft_consensus.cc:348] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.912716 3288 raft_consensus.cc:374] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.912762 3290 raft_consensus.cc:374] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.912842 3288 raft_consensus.cc:729] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.912884 3286 raft_consensus.cc:374] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.912899 3290 raft_consensus.cc:729] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.913015 3286 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.913741 3288 consensus_queue.cc:260] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.913743 3286 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.913749 3290 consensus_queue.cc:260] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.915531 3287 raft_consensus.cc:348] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.916112 3287 raft_consensus.cc:374] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.916261 3287 raft_consensus.cc:729] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.917239 3287 consensus_queue.cc:260] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.922349 3286 ts_tablet_manager.cc:1398] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20240503 14:15:36.922983 3286 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:36.925508 3286 tablet_bootstrap.cc:654] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.927277 3290 ts_tablet_manager.cc:1398] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.024s user 0.019s sys 0.004s
I20240503 14:15:36.927893 3290 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:36.930224 3290 tablet_bootstrap.cc:654] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.933025 3288 ts_tablet_manager.cc:1398] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.030s user 0.014s sys 0.004s
I20240503 14:15:36.934777 3286 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:36.935038 3286 ts_tablet_manager.cc:1367] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:15:36.935483 3288 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
W20240503 14:15:36.936477 2608 tablet.cc:2369] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:36.937738 3286 raft_consensus.cc:348] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.938110 3288 tablet_bootstrap.cc:654] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.938138 3286 raft_consensus.cc:374] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.938251 3286 raft_consensus.cc:729] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.938669 3286 consensus_queue.cc:260] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.939612 3286 ts_tablet_manager.cc:1398] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:36.941210 3290 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:36.941397 3290 ts_tablet_manager.cc:1367] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20240503 14:15:36.942735 3290 raft_consensus.cc:348] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.942999 3287 ts_tablet_manager.cc:1398] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.036s user 0.020s sys 0.006s
I20240503 14:15:36.943092 3290 raft_consensus.cc:374] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.943269 3290 raft_consensus.cc:729] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.943579 3290 consensus_queue.cc:260] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.943923 3287 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:36.944448 3290 ts_tablet_manager.cc:1398] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:36.946930 3287 tablet_bootstrap.cc:654] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:36.948972 3288 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:36.949113 3288 ts_tablet_manager.cc:1367] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240503 14:15:36.949342 3287 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:36.949472 3287 ts_tablet_manager.cc:1367] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:15:36.949949 3288 raft_consensus.cc:348] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.950101 3288 raft_consensus.cc:374] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.950179 3288 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.950222 3287 raft_consensus.cc:348] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.950439 3287 raft_consensus.cc:374] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.950398 3288 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.950537 3287 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.950767 3287 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.951222 3287 ts_tablet_manager.cc:1398] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:36.951411 3288 ts_tablet_manager.cc:1398] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:36.951733 3288 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:36.953399 3288 tablet_bootstrap.cc:654] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
W20240503 14:15:36.953351 2478 tablet.cc:2369] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:36.955622 3288 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:36.955807 3288 ts_tablet_manager.cc:1367] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:15:36.956467 3288 raft_consensus.cc:348] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.956642 3288 raft_consensus.cc:374] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:36.956696 3288 raft_consensus.cc:729] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.956912 3288 consensus_queue.cc:260] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:36.957356 3288 ts_tablet_manager.cc:1398] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240503 14:15:36.980294 2347 tablet.cc:2369] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:36.989162 3296 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:36.989405 3296 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:36.991084 3296 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:36.998947 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:36.999233 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:36.999477 2432 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:36.999701 2692 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:37.000144 2235 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:37.000537 3296 raft_consensus.cc:2781] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:37.000691 3296 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:37.000802 3296 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.002780 3296 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:37.003463 3296 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:37.003978 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:37.004145 2692 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.006183 2692 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:37.006484 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:37.006603 2234 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:37.006690 2432 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.006907 3296 raft_consensus.cc:2781] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:37.007351 3296 raft_consensus.cc:686] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:37.007894 3296 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:37.009135 2432 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:37.012225 2159 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
W20240503 14:15:37.172950 2738 tablet.cc:2369] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:37.209582 3297 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:37.209789 3297 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:37.211447 3297 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:37.219401 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:37.219481 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:37.219834 2301 raft_consensus.cc:2454] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:37.220016 2562 raft_consensus.cc:2454] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:37.220398 2626 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:37.220561 3294 raft_consensus.cc:482] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:37.220752 3294 raft_consensus.cc:504] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:37.221053 3297 raft_consensus.cc:2781] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:37.221175 3297 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:37.221274 3297 raft_consensus.cc:3037] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.222342 3294 leader_election.cc:288] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:37.224181 3297 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:37.224893 3297 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:37.225559 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:37.225750 2301 raft_consensus.cc:3037] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.225936 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:37.226174 2562 raft_consensus.cc:3037] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.228847 2301 raft_consensus.cc:2454] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:37.229257 2626 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:37.229611 3297 raft_consensus.cc:2781] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:37.230049 2562 raft_consensus.cc:2454] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:37.233207 3297 raft_consensus.cc:686] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:37.233809 3297 consensus_queue.cc:237] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:37.240515 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c94404d8d4f45d59958ae6feb36fc24" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:37.240759 2301 raft_consensus.cc:2454] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:37.241519 2159 catalog_manager.cc:5484] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:37.242810 2366 leader_election.cc:302] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:37.243796 3294 raft_consensus.cc:2781] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:37.243913 3294 raft_consensus.cc:482] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:37.243988 3294 raft_consensus.cc:3037] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.244553 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c94404d8d4f45d59958ae6feb36fc24" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:37.244802 2562 raft_consensus.cc:2454] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:37.246909 3294 raft_consensus.cc:504] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:37.247608 3294 leader_election.cc:288] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:37.248037 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c94404d8d4f45d59958ae6feb36fc24" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:37.248040 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c94404d8d4f45d59958ae6feb36fc24" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:37.248235 2562 raft_consensus.cc:3037] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.248227 2301 raft_consensus.cc:3037] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:37.250012 2562 raft_consensus.cc:2454] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:37.250015 2301 raft_consensus.cc:2454] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:37.250458 2365 leader_election.cc:302] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:37.250767 3294 raft_consensus.cc:2781] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:37.251166 3294 raft_consensus.cc:686] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:37.251706 3294 consensus_queue.cc:237] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:37.255772 2159 catalog_manager.cc:5484] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:37.436831 3296 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:37.443444 3305 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:37.653617 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:37.654047 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:15:37.661041 3294 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:37.663129 3315 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:37.673331 3294 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 14:15:37.724094 3297 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:37.729663 3297 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 263.797 ms
time per row: 0.131899 ms
I20240503 14:15:38.084945 2158 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:38.093824 2412 tablet_service.cc:1467] Processing CreateTablet for tablet eb1d30b76c4d43d0a54a49892d09a6f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.093906 2541 tablet_service.cc:1467] Processing CreateTablet for tablet eb1d30b76c4d43d0a54a49892d09a6f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.094343 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1d30b76c4d43d0a54a49892d09a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.094341 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1d30b76c4d43d0a54a49892d09a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.094282 2672 tablet_service.cc:1467] Processing CreateTablet for tablet eb1d30b76c4d43d0a54a49892d09a6f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.094695 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1d30b76c4d43d0a54a49892d09a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.096031 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 944d2a983d6247498b02a6e7b7daf746 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.096486 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d2a983d6247498b02a6e7b7daf746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.096769 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 9974d80e8d31417788e10140de97e722 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.097142 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9974d80e8d31417788e10140de97e722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.099645 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 944d2a983d6247498b02a6e7b7daf746 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.100052 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d2a983d6247498b02a6e7b7daf746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.106477 3379 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:38.106451 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 9974d80e8d31417788e10140de97e722 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.107057 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9974d80e8d31417788e10140de97e722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.108692 3379 tablet_bootstrap.cc:654] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.111829 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 944d2a983d6247498b02a6e7b7daf746 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.112365 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d2a983d6247498b02a6e7b7daf746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.113870 3381 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:38.114802 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 9974d80e8d31417788e10140de97e722 (DEFAULT_TABLE table=rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:38.115345 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9974d80e8d31417788e10140de97e722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:38.116178 3381 tablet_bootstrap.cc:654] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.117542 3379 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:38.117740 3379 ts_tablet_manager.cc:1367] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 14:15:38.118835 3379 raft_consensus.cc:348] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.119165 3379 raft_consensus.cc:374] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.119309 3379 raft_consensus.cc:729] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.119599 3379 consensus_queue.cc:260] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.120337 3379 ts_tablet_manager.cc:1398] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:15:38.136020 3379 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:38.137760 3379 tablet_bootstrap.cc:654] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.142372 3383 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:38.145051 3383 tablet_bootstrap.cc:654] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.148296 3378 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:38.150260 3378 tablet_bootstrap.cc:654] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.150267 3383 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:38.150414 3383 ts_tablet_manager.cc:1367] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 14:15:38.150717 3381 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:38.150888 3381 ts_tablet_manager.cc:1367] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.010s
I20240503 14:15:38.151131 3383 raft_consensus.cc:348] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.151439 3383 raft_consensus.cc:374] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.151683 3383 raft_consensus.cc:729] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.151827 3381 raft_consensus.cc:348] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.152040 3381 raft_consensus.cc:374] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.151965 3383 consensus_queue.cc:260] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.152146 3381 raft_consensus.cc:729] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.152344 3379 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:38.152523 3379 ts_tablet_manager.cc:1367] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240503 14:15:38.152448 3381 consensus_queue.cc:260] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.153025 3381 ts_tablet_manager.cc:1398] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:38.153349 3381 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:38.154212 3379 raft_consensus.cc:348] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.154469 3379 raft_consensus.cc:374] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.154552 3379 raft_consensus.cc:729] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.154754 3383 ts_tablet_manager.cc:1398] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:38.155241 3381 tablet_bootstrap.cc:654] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.155817 3379 consensus_queue.cc:260] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.156548 3378 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:38.156734 3378 ts_tablet_manager.cc:1367] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240503 14:15:38.157594 3379 ts_tablet_manager.cc:1398] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:15:38.158643 3378 raft_consensus.cc:348] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.158833 3378 raft_consensus.cc:374] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.158923 3378 raft_consensus.cc:729] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.159173 3378 consensus_queue.cc:260] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.159601 3381 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:38.159762 3381 ts_tablet_manager.cc:1367] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 14:15:38.159919 3378 ts_tablet_manager.cc:1398] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:38.160362 3378 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:38.160535 3381 raft_consensus.cc:348] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.160714 3381 raft_consensus.cc:374] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.160796 3381 raft_consensus.cc:729] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.161043 3381 consensus_queue.cc:260] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.161532 3381 ts_tablet_manager.cc:1398] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:38.161823 3381 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:38.161984 3378 tablet_bootstrap.cc:654] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.163228 3381 tablet_bootstrap.cc:654] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.165328 3378 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:38.165478 3378 ts_tablet_manager.cc:1367] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:15:38.165921 3381 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:38.166069 3381 ts_tablet_manager.cc:1367] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:15:38.166196 3378 raft_consensus.cc:348] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.166414 3378 raft_consensus.cc:374] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.166498 3378 raft_consensus.cc:729] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.166667 3381 raft_consensus.cc:348] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.166819 3381 raft_consensus.cc:374] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.166746 3378 consensus_queue.cc:260] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.166903 3381 raft_consensus.cc:729] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.167111 3381 consensus_queue.cc:260] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.167270 3378 ts_tablet_manager.cc:1398] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:38.167557 3378 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:38.167913 3381 ts_tablet_manager.cc:1398] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:38.169550 3378 tablet_bootstrap.cc:654] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:38.171564 3378 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:38.171720 3378 ts_tablet_manager.cc:1367] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:15:38.172304 3378 raft_consensus.cc:348] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.172461 3378 raft_consensus.cc:374] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:38.172559 3378 raft_consensus.cc:729] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:38.172752 3378 consensus_queue.cc:260] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.173152 3378 ts_tablet_manager.cc:1398] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:38.216118 3311 raft_consensus.cc:482] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:38.216307 3311 raft_consensus.cc:504] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.217195 3311 leader_election.cc:288] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:38.217947 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9974d80e8d31417788e10140de97e722" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:38.218168 2562 raft_consensus.cc:2454] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:38.218613 2625 leader_election.cc:302] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:38.218976 3311 raft_consensus.cc:2781] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:38.219141 3311 raft_consensus.cc:482] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:38.219290 3311 raft_consensus.cc:3037] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.221833 3311 raft_consensus.cc:504] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.222561 3311 leader_election.cc:288] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:38.223311 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9974d80e8d31417788e10140de97e722" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:38.223515 2562 raft_consensus.cc:3037] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.225095 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9974d80e8d31417788e10140de97e722" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:38.225297 2432 raft_consensus.cc:3037] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.225035 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9974d80e8d31417788e10140de97e722" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:38.225973 2562 raft_consensus.cc:2454] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:38.226405 2625 leader_election.cc:302] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:38.226725 3311 raft_consensus.cc:2781] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:38.227223 3311 raft_consensus.cc:686] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:38.227524 3311 consensus_queue.cc:237] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.228040 2432 raft_consensus.cc:2454] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:38.230495 2154 catalog_manager.cc:5484] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:38.333966 3296 raft_consensus.cc:482] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:38.334172 3296 raft_consensus.cc:504] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.334954 3296 leader_election.cc:288] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:38.335587 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d2a983d6247498b02a6e7b7daf746" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:38.335826 2432 raft_consensus.cc:2454] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:38.336256 2235 leader_election.cc:302] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:38.336552 3296 raft_consensus.cc:2781] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:38.336740 3296 raft_consensus.cc:482] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:38.336855 3296 raft_consensus.cc:3037] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.339543 3296 raft_consensus.cc:504] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.340157 3296 leader_election.cc:288] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:38.340713 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d2a983d6247498b02a6e7b7daf746" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:38.340945 2432 raft_consensus.cc:3037] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.342494 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d2a983d6247498b02a6e7b7daf746" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:38.342742 2562 raft_consensus.cc:2454] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:38.342640 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944d2a983d6247498b02a6e7b7daf746" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:38.342892 2561 raft_consensus.cc:3037] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.343606 2432 raft_consensus.cc:2454] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:38.343983 2235 leader_election.cc:302] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:38.344336 3371 raft_consensus.cc:2781] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:38.344514 3371 raft_consensus.cc:686] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:38.344846 3371 consensus_queue.cc:237] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:38.345415 2561 raft_consensus.cc:2454] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:38.348069 2154 catalog_manager.cc:5484] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:15:38.435393 3311 raft_consensus.cc:482] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:38.435618 3311 raft_consensus.cc:504] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.436196 3311 leader_election.cc:288] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:38.436725 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:38.436741 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:38.436959 2562 raft_consensus.cc:2454] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:38.436975 2432 raft_consensus.cc:2454] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:38.437367 2626 leader_election.cc:302] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:38.437649 3311 raft_consensus.cc:2781] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:38.437743 3311 raft_consensus.cc:482] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:38.437832 3311 raft_consensus.cc:3037] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.439723 3311 raft_consensus.cc:504] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.440323 3311 leader_election.cc:288] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:38.440791 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:38.440768 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:38.441009 2562 raft_consensus.cc:3037] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.441018 2432 raft_consensus.cc:3037] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:38.443820 2562 raft_consensus.cc:2454] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:38.443833 2432 raft_consensus.cc:2454] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:38.444234 2625 leader_election.cc:302] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:38.444510 3311 raft_consensus.cc:2781] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:38.444665 3311 raft_consensus.cc:686] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:38.444888 3311 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:38.447265 2154 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:38.705166 3311 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:38.710484 3297 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:38.816679 3371 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:38.823309 3371 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:38.836997 3297 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:38.841732 3297 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:38.871877 3397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:38.872184 3397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:38.879604 3397 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
W20240503 14:15:39.103351 2626 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 54436.5us
INSERT report
rows total: 2000
time total: 258.047 ms
time per row: 0.129023 ms
I20240503 14:15:39.280164 2154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:39.288476 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf4675882e64f35bc3dc9ee06ae366d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.288661 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf4675882e64f35bc3dc9ee06ae366d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.288905 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf4675882e64f35bc3dc9ee06ae366d (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.289017 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.289031 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.289336 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.292500 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 1d584b7026b64ba1b720ed21d7019740 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.292668 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 75bafc7675ca45639b696db6e4b7ce36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.292927 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d584b7026b64ba1b720ed21d7019740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.293079 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bafc7675ca45639b696db6e4b7ce36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.295939 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 1d584b7026b64ba1b720ed21d7019740 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.296384 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d584b7026b64ba1b720ed21d7019740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.299566 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 75bafc7675ca45639b696db6e4b7ce36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.300098 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bafc7675ca45639b696db6e4b7ce36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.300117 2672 tablet_service.cc:1467] Processing CreateTablet for tablet 1d584b7026b64ba1b720ed21d7019740 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.300549 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d584b7026b64ba1b720ed21d7019740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.303874 2670 tablet_service.cc:1467] Processing CreateTablet for tablet 75bafc7675ca45639b696db6e4b7ce36 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:39.305470 3465 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:39.307545 3465 tablet_bootstrap.cc:654] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.307562 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bafc7675ca45639b696db6e4b7ce36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:39.330263 3467 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:39.333292 3467 tablet_bootstrap.cc:654] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.335083 3464 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:39.336133 3463 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:39.336689 3467 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:39.336831 3465 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:39.336846 3467 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:15:39.336993 3465 ts_tablet_manager.cc:1367] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.018s
I20240503 14:15:39.337006 3464 tablet_bootstrap.cc:654] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.337672 3467 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.337813 3465 raft_consensus.cc:348] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.337993 3465 raft_consensus.cc:374] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.338093 3465 raft_consensus.cc:729] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.338346 3465 consensus_queue.cc:260] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.338660 3467 raft_consensus.cc:374] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.339071 3465 ts_tablet_manager.cc:1398] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:39.339047 3467 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.339361 3467 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.340010 3467 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:39.340382 3467 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:39.341361 3463 tablet_bootstrap.cc:654] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.342039 3467 tablet_bootstrap.cc:654] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.346565 3467 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:39.347704 3467 ts_tablet_manager.cc:1367] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240503 14:15:39.348160 3464 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:39.348312 3464 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 14:15:39.348915 3464 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.348812 3467 raft_consensus.cc:348] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.349073 3464 raft_consensus.cc:374] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.349108 3467 raft_consensus.cc:374] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.349184 3464 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.349210 3467 raft_consensus.cc:729] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.349478 3464 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.349577 3467 consensus_queue.cc:260] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.350476 3467 ts_tablet_manager.cc:1398] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:39.350546 3464 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:39.350898 3464 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:39.350893 3467 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:39.352595 3464 tablet_bootstrap.cc:654] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.353178 3467 tablet_bootstrap.cc:654] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.355537 3463 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:39.355581 3467 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:39.355706 3467 ts_tablet_manager.cc:1367] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:15:39.355777 3463 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.010s
I20240503 14:15:39.356575 3467 raft_consensus.cc:348] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.356583 3463 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.356822 3467 raft_consensus.cc:374] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.356921 3463 raft_consensus.cc:374] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.356927 3467 raft_consensus.cc:729] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.357208 3467 consensus_queue.cc:260] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.357070 3463 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.357826 3467 ts_tablet_manager.cc:1398] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:39.357815 3463 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.358614 3463 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:39.358939 3463 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:39.361176 3463 tablet_bootstrap.cc:654] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.361724 3464 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:39.361944 3464 ts_tablet_manager.cc:1367] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 14:15:39.362767 3464 raft_consensus.cc:348] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.363039 3464 raft_consensus.cc:374] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.363114 3464 raft_consensus.cc:729] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.363389 3464 consensus_queue.cc:260] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.363999 3463 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:39.363986 3464 ts_tablet_manager.cc:1398] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:39.364197 3463 ts_tablet_manager.cc:1367] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:39.364302 3464 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:39.365010 3463 raft_consensus.cc:348] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.365264 3463 raft_consensus.cc:374] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.365374 3463 raft_consensus.cc:729] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.365638 3463 consensus_queue.cc:260] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.365939 3464 tablet_bootstrap.cc:654] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:39.366261 3463 ts_tablet_manager.cc:1398] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:39.368424 3464 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:39.368587 3464 ts_tablet_manager.cc:1367] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 14:15:39.369181 3464 raft_consensus.cc:348] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.369324 3464 raft_consensus.cc:374] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:39.369405 3464 raft_consensus.cc:729] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:39.369642 3464 consensus_queue.cc:260] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.370151 3464 ts_tablet_manager.cc:1398] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:15:39.442092 3371 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.442314 3371 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.442945 3371 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:39.443497 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:39.443619 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:39.443797 2692 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:39.443837 2562 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:39.444253 2234 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:39.444562 3371 raft_consensus.cc:2781] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:39.444666 3371 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:39.444762 3371 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.446614 3371 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.447149 3371 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:39.447638 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:39.447728 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:39.447824 2692 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.447937 2562 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.449671 2692 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:39.449798 2562 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:39.450158 2234 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:39.450462 3371 raft_consensus.cc:2781] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:39.450615 3371 raft_consensus.cc:686] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:39.450902 3371 consensus_queue.cc:237] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:39.453522 2153 catalog_manager.cc:5484] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:15:39.497453 3296 raft_consensus.cc:482] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.497668 3296 raft_consensus.cc:504] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.498508 3296 leader_election.cc:288] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:39.498847 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:39.498894 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:39.499068 2692 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:39.499109 2431 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:39.499502 2234 leader_election.cc:302] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:39.499789 3296 raft_consensus.cc:2781] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:39.499902 3296 raft_consensus.cc:482] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:39.500001 3296 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.501814 3296 raft_consensus.cc:504] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.502442 3296 leader_election.cc:288] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:39.502825 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:39.502857 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:39.503005 2692 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.503026 2431 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.504868 2692 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:39.504940 2431 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:39.505256 2234 leader_election.cc:302] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:39.505548 3296 raft_consensus.cc:2781] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:39.505713 3296 raft_consensus.cc:686] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:39.505982 3296 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.508631 2153 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:15:39.631559 3296 raft_consensus.cc:482] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:39.631779 3296 raft_consensus.cc:504] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.632426 3296 leader_election.cc:288] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:39.633019 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:39.633090 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:39.633240 2692 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:39.633332 2431 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:39.633653 2234 leader_election.cc:302] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:39.633960 3296 raft_consensus.cc:2781] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:39.634045 3296 raft_consensus.cc:482] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:39.634150 3296 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.636024 3296 raft_consensus.cc:504] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.636600 3296 leader_election.cc:288] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:39.636991 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:39.637239 2692 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.637243 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:39.637460 2431 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:39.640187 2692 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:39.640266 2431 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:39.640674 2234 leader_election.cc:302] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:39.640969 3296 raft_consensus.cc:2781] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:39.641136 3296 raft_consensus.cc:686] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:39.641440 3296 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:39.644027 2153 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:15:39.838332 3296 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.844082 3296 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.969429 3296 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:39.974910 3371 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:40.030463 3479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:40.030777 3479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:40.038349 3479 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:15:40.167220 3371 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:40.236954 2691 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:40.238241 3518 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:40.269721 3485 mvcc.cc:205] Tried to move back new op lower bound from 7023598551950336000 to 7023598549573652480. Current Snapshot: MvccSnapshot[applied={T|T < 7023598551950336000}]
I20240503 14:15:40.275717 3517 mvcc.cc:205] Tried to move back new op lower bound from 7023598551950336000 to 7023598549573652480. Current Snapshot: MvccSnapshot[applied={T|T < 7023598551950336000 or (T in {7023598551950336000})}]
INSERT report
rows total: 2000
time total: 201.462 ms
time per row: 0.100731 ms
I20240503 14:15:40.381434 2152 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:40.390302 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 470b425f5cef494ab3334fe2b85e1c4e (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.390497 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 470b425f5cef494ab3334fe2b85e1c4e (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.390815 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b425f5cef494ab3334fe2b85e1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.390971 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b425f5cef494ab3334fe2b85e1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.392144 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 61b63f87fb384bb2a48044bcad87c282 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.392180 2541 tablet_service.cc:1467] Processing CreateTablet for tablet fda4460865854505b5220d7ec3b34c57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.392563 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b63f87fb384bb2a48044bcad87c282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.393441 2279 tablet_service.cc:1467] Processing CreateTablet for tablet fda4460865854505b5220d7ec3b34c57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.393807 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4460865854505b5220d7ec3b34c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.395618 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 470b425f5cef494ab3334fe2b85e1c4e (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.396078 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b425f5cef494ab3334fe2b85e1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.397285 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 61b63f87fb384bb2a48044bcad87c282 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.397683 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b63f87fb384bb2a48044bcad87c282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.399900 2671 tablet_service.cc:1467] Processing CreateTablet for tablet fda4460865854505b5220d7ec3b34c57 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.400086 2670 tablet_service.cc:1467] Processing CreateTablet for tablet 61b63f87fb384bb2a48044bcad87c282 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:40.400322 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4460865854505b5220d7ec3b34c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.400527 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b63f87fb384bb2a48044bcad87c282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.403965 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4460865854505b5220d7ec3b34c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:40.409823 3541 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:40.411226 3542 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:40.412129 3541 tablet_bootstrap.cc:654] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.415596 3542 tablet_bootstrap.cc:654] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.415975 3540 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:40.416322 3539 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:40.417801 3541 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:40.417963 3541 ts_tablet_manager.cc:1367] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 14:15:40.418222 3539 tablet_bootstrap.cc:654] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.418468 3540 tablet_bootstrap.cc:654] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.418871 3541 raft_consensus.cc:348] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.419205 3541 raft_consensus.cc:374] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.419353 3541 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.419694 3541 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.420521 3539 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:40.420604 3541 ts_tablet_manager.cc:1398] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:15:40.420756 3539 ts_tablet_manager.cc:1367] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 14:15:40.420979 3541 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:40.421622 3539 raft_consensus.cc:348] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.421871 3539 raft_consensus.cc:374] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.421981 3539 raft_consensus.cc:729] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.422245 3539 consensus_queue.cc:260] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.422848 3539 ts_tablet_manager.cc:1398] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:40.423245 3539 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:40.424422 3541 tablet_bootstrap.cc:654] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.424806 3539 tablet_bootstrap.cc:654] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.426577 3540 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:40.426775 3540 ts_tablet_manager.cc:1367] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:15:40.427711 3540 raft_consensus.cc:348] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.427973 3540 raft_consensus.cc:374] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.428102 3540 raft_consensus.cc:729] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.428366 3540 consensus_queue.cc:260] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.428973 3540 ts_tablet_manager.cc:1398] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.438838 3540 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:40.441387 3540 tablet_bootstrap.cc:654] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.445407 3539 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:40.445667 3539 ts_tablet_manager.cc:1367] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.011s
I20240503 14:15:40.446072 3540 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:40.446288 3540 ts_tablet_manager.cc:1367] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:15:40.446369 3542 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:40.446597 3542 ts_tablet_manager.cc:1367] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.016s
I20240503 14:15:40.446609 3539 raft_consensus.cc:348] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.446841 3539 raft_consensus.cc:374] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.446926 3539 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.447221 3540 raft_consensus.cc:348] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.447230 3539 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.447520 3540 raft_consensus.cc:374] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.447434 3542 raft_consensus.cc:348] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.447628 3542 raft_consensus.cc:374] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.447682 3540 raft_consensus.cc:729] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.447785 3542 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.447932 3539 ts_tablet_manager.cc:1398] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.448084 3542 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.448627 3541 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:40.448850 3541 ts_tablet_manager.cc:1367] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.018s
I20240503 14:15:40.449761 3542 ts_tablet_manager.cc:1398] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:40.449672 3541 raft_consensus.cc:348] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.449901 3541 raft_consensus.cc:374] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.450026 3541 raft_consensus.cc:729] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.450160 3542 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:40.450320 3541 consensus_queue.cc:260] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.451013 3541 ts_tablet_manager.cc:1398] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:40.448032 3540 consensus_queue.cc:260] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.452085 3542 tablet_bootstrap.cc:654] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.452374 3540 ts_tablet_manager.cc:1398] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:40.456305 3542 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:40.456470 3542 ts_tablet_manager.cc:1367] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:15:40.457114 3542 raft_consensus.cc:348] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.457334 3542 raft_consensus.cc:374] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.457474 3542 raft_consensus.cc:729] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.457698 3542 consensus_queue.cc:260] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.458159 3542 ts_tablet_manager.cc:1398] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:40.458488 3542 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:40.459801 3542 tablet_bootstrap.cc:654] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:40.461687 3542 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:40.461829 3542 ts_tablet_manager.cc:1367] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:40.462476 3542 raft_consensus.cc:348] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.462656 3542 raft_consensus.cc:374] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:40.462756 3542 raft_consensus.cc:729] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:40.462958 3542 consensus_queue.cc:260] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.463490 3542 ts_tablet_manager.cc:1398] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:40.626559 3297 raft_consensus.cc:482] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.626747 3297 raft_consensus.cc:504] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.627471 3297 leader_election.cc:288] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:40.628010 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b63f87fb384bb2a48044bcad87c282" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:40.628017 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b63f87fb384bb2a48044bcad87c282" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:40.628245 2562 raft_consensus.cc:2454] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:40.628263 2431 raft_consensus.cc:2454] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:40.628732 2626 leader_election.cc:302] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:40.629040 3297 raft_consensus.cc:2781] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:40.629161 3297 raft_consensus.cc:482] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:40.629256 3297 raft_consensus.cc:3037] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.631081 3297 raft_consensus.cc:504] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.631664 3297 leader_election.cc:288] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:40.632134 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b63f87fb384bb2a48044bcad87c282" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:40.632172 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b63f87fb384bb2a48044bcad87c282" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:40.632341 2431 raft_consensus.cc:3037] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.632372 2562 raft_consensus.cc:3037] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.634292 2431 raft_consensus.cc:2454] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:40.634341 2562 raft_consensus.cc:2454] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:40.634706 2626 leader_election.cc:302] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:40.634982 3297 raft_consensus.cc:2781] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:40.635134 3297 raft_consensus.cc:686] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:40.635450 3297 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:40.638013 2154 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:40.639667 3297 raft_consensus.cc:482] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.639819 3297 raft_consensus.cc:504] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.640348 3297 leader_election.cc:288] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:40.640947 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:40.641175 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:40.641237 2562 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:40.641395 2301 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:40.641688 2625 leader_election.cc:302] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:40.641988 3297 raft_consensus.cc:2781] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:40.642083 3297 raft_consensus.cc:482] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:40.642192 3297 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.644095 3297 raft_consensus.cc:504] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.644680 3297 leader_election.cc:288] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:40.645079 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:40.645146 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:40.645236 2562 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.645337 2301 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.647049 2562 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:40.647295 2301 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:40.647455 2625 leader_election.cc:302] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:40.647801 3297 raft_consensus.cc:2781] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:40.647966 3297 raft_consensus.cc:686] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:40.648219 3297 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:40.650756 2154 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:15:40.744978 3294 raft_consensus.cc:482] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:40.745187 3294 raft_consensus.cc:504] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.745786 3294 leader_election.cc:288] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:40.746371 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b425f5cef494ab3334fe2b85e1c4e" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:40.746373 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b425f5cef494ab3334fe2b85e1c4e" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:40.746596 2301 raft_consensus.cc:2454] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:40.746634 2562 raft_consensus.cc:2454] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:40.746986 2366 leader_election.cc:302] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:40.747326 3294 raft_consensus.cc:2781] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:40.747455 3294 raft_consensus.cc:482] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:40.747570 3294 raft_consensus.cc:3037] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.749475 3294 raft_consensus.cc:504] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.750082 3294 leader_election.cc:288] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:40.750458 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b425f5cef494ab3334fe2b85e1c4e" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:40.750655 2562 raft_consensus.cc:3037] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.750595 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470b425f5cef494ab3334fe2b85e1c4e" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:40.750800 2301 raft_consensus.cc:3037] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:40.752628 2562 raft_consensus.cc:2454] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:40.752656 2301 raft_consensus.cc:2454] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:40.753050 2366 leader_election.cc:302] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:40.753312 3294 raft_consensus.cc:2781] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:40.753770 3294 raft_consensus.cc:686] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:40.754009 3294 consensus_queue.cc:237] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:40.756601 2154 catalog_manager.cc:5484] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:15:41.125344 3297 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.130568 3553 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:41.131717 3558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:41.132030 3558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:15:41.139580 3553 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:41.141891 3558 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:41.144027 3553 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240503 14:15:41.258808 3294 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:41.274008 3294 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 291.517 ms
time per row: 0.145758 ms
I20240503 14:15:41.586371 2158 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:41.595441 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 65a196c723884d068b3728e876c87eb2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.595546 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 65a196c723884d068b3728e876c87eb2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.595896 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a196c723884d068b3728e876c87eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.596009 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a196c723884d068b3728e876c87eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.597184 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 65a196c723884d068b3728e876c87eb2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.597381 2671 tablet_service.cc:1467] Processing CreateTablet for tablet af99824d1853471b867e4ff1f01e9367 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.597592 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a196c723884d068b3728e876c87eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.597800 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af99824d1853471b867e4ff1f01e9367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.602373 2541 tablet_service.cc:1467] Processing CreateTablet for tablet f139d77b0f0c47809e6408b2771adb73 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.602752 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f139d77b0f0c47809e6408b2771adb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.603559 2670 tablet_service.cc:1467] Processing CreateTablet for tablet f139d77b0f0c47809e6408b2771adb73 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.603926 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f139d77b0f0c47809e6408b2771adb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.607982 2410 tablet_service.cc:1467] Processing CreateTablet for tablet af99824d1853471b867e4ff1f01e9367 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.608232 2279 tablet_service.cc:1467] Processing CreateTablet for tablet af99824d1853471b867e4ff1f01e9367 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.608392 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af99824d1853471b867e4ff1f01e9367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.608619 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af99824d1853471b867e4ff1f01e9367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.613127 3629 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:41.613840 3627 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:41.615566 2280 tablet_service.cc:1467] Processing CreateTablet for tablet f139d77b0f0c47809e6408b2771adb73 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:41.615871 3627 tablet_bootstrap.cc:654] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.615923 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f139d77b0f0c47809e6408b2771adb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:41.623381 3628 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:41.624275 3627 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:41.624439 3627 ts_tablet_manager.cc:1367] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 14:15:41.625298 3628 tablet_bootstrap.cc:654] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.625281 3627 raft_consensus.cc:348] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.625553 3627 raft_consensus.cc:374] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.625674 3627 raft_consensus.cc:729] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.625923 3627 consensus_queue.cc:260] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.626681 3627 ts_tablet_manager.cc:1398] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:41.627079 3627 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:41.629242 3627 tablet_bootstrap.cc:654] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.631542 3628 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:41.631758 3628 ts_tablet_manager.cc:1367] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:15:41.633009 3628 raft_consensus.cc:348] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.633221 3628 raft_consensus.cc:374] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.633360 3628 raft_consensus.cc:729] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.633648 3628 consensus_queue.cc:260] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.634162 3629 tablet_bootstrap.cc:654] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.634248 3628 ts_tablet_manager.cc:1398] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:41.634562 3628 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:41.635300 3626 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:41.636749 3628 tablet_bootstrap.cc:654] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.637324 3626 tablet_bootstrap.cc:654] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.645588 3628 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:41.645944 3628 ts_tablet_manager.cc:1367] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 14:15:41.647110 3628 raft_consensus.cc:348] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.647531 3629 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:41.647547 3628 raft_consensus.cc:374] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.647622 3628 raft_consensus.cc:729] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.648001 3628 consensus_queue.cc:260] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.648502 3627 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:41.648713 3627 ts_tablet_manager.cc:1367] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20240503 14:15:41.649303 3628 ts_tablet_manager.cc:1398] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:41.649570 3627 raft_consensus.cc:348] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.649885 3627 raft_consensus.cc:374] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.649924 3591 raft_consensus.cc:482] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:41.650003 3627 raft_consensus.cc:729] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.650099 3591 raft_consensus.cc:504] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.650285 3627 consensus_queue.cc:260] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.650867 3626 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:41.650933 3591 leader_election.cc:288] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:41.651046 3626 ts_tablet_manager.cc:1367] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 14:15:41.651518 3627 ts_tablet_manager.cc:1398] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:41.652428 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a196c723884d068b3728e876c87eb2" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:41.653582 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a196c723884d068b3728e876c87eb2" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:41.654012 2562 raft_consensus.cc:2454] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:41.654994 2365 leader_election.cc:302] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:41.656152 3591 raft_consensus.cc:2781] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:41.656284 3591 raft_consensus.cc:482] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:41.656383 3591 raft_consensus.cc:3037] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:15:41.656659 2366 leader_election.cc:341] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:41.660037 3591 raft_consensus.cc:504] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.660668 3626 raft_consensus.cc:348] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.660893 3626 raft_consensus.cc:374] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.660980 3626 raft_consensus.cc:729] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.661212 3626 consensus_queue.cc:260] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.661464 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a196c723884d068b3728e876c87eb2" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:41.661700 2301 raft_consensus.cc:3037] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.661852 3626 ts_tablet_manager.cc:1398] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240503 14:15:41.662155 3626 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:41.663980 3626 tablet_bootstrap.cc:654] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.664032 3591 leader_election.cc:288] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:41.664556 2301 raft_consensus.cc:2454] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:41.647802 3629 ts_tablet_manager.cc:1367] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.005s
I20240503 14:15:41.665004 2366 leader_election.cc:302] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:41.665488 3591 raft_consensus.cc:2781] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:41.665711 3591 raft_consensus.cc:686] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:41.665800 3629 raft_consensus.cc:348] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.665999 3629 raft_consensus.cc:374] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.666096 3629 raft_consensus.cc:729] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.665978 3591 consensus_queue.cc:237] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:41.666406 3629 consensus_queue.cc:260] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.667093 3629 ts_tablet_manager.cc:1398] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:41.667533 3629 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:41.669308 3629 tablet_bootstrap.cc:654] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.671756 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a196c723884d068b3728e876c87eb2" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:41.671967 2562 raft_consensus.cc:3037] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.674890 3626 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:41.675241 3626 ts_tablet_manager.cc:1367] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240503 14:15:41.675441 2562 raft_consensus.cc:2454] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:41.676209 3626 raft_consensus.cc:348] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.676425 3626 raft_consensus.cc:374] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.676496 3626 raft_consensus.cc:729] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.676769 3626 consensus_queue.cc:260] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.677646 3629 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:41.677878 3629 ts_tablet_manager.cc:1367] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240503 14:15:41.677917 3626 ts_tablet_manager.cc:1398] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:41.678189 3626 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:41.678851 3629 raft_consensus.cc:348] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.679279 3629 raft_consensus.cc:374] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.679428 3629 raft_consensus.cc:729] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.679749 3629 consensus_queue.cc:260] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.680033 3626 tablet_bootstrap.cc:654] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:41.680405 3629 ts_tablet_manager.cc:1398] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:41.682111 3531 raft_consensus.cc:482] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:41.682255 3531 raft_consensus.cc:504] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.683015 3531 leader_election.cc:288] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:41.683454 3592 raft_consensus.cc:482] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:41.683642 3592 raft_consensus.cc:504] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.684085 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af99824d1853471b867e4ff1f01e9367" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:41.684361 2691 raft_consensus.cc:2454] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:41.684388 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af99824d1853471b867e4ff1f01e9367" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:41.684563 3592 leader_election.cc:288] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:41.684571 2432 raft_consensus.cc:2454] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:41.685214 2234 leader_election.cc:302] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:41.686368 3531 raft_consensus.cc:2781] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:41.686554 3531 raft_consensus.cc:482] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:41.686659 3531 raft_consensus.cc:3037] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.687865 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af99824d1853471b867e4ff1f01e9367" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:41.690210 3626 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:41.690351 3626 ts_tablet_manager.cc:1367] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 14:15:41.691098 3626 raft_consensus.cc:348] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.691313 3626 raft_consensus.cc:374] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:41.691421 3626 raft_consensus.cc:729] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:41.691632 3626 consensus_queue.cc:260] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.691859 3531 raft_consensus.cc:504] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.692441 2301 raft_consensus.cc:2379] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 685c6181561b4056abdecfd6d8bdb3b2 in current term 1: Already voted for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in this term.
I20240503 14:15:41.694685 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af99824d1853471b867e4ff1f01e9367" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:41.694978 2432 raft_consensus.cc:3037] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.694909 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af99824d1853471b867e4ff1f01e9367" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:41.695163 2691 raft_consensus.cc:3037] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.698129 2691 raft_consensus.cc:2454] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:41.698657 2234 leader_election.cc:302] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:41.698959 2432 raft_consensus.cc:2454] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:41.700166 3531 leader_election.cc:288] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:41.701370 3531 raft_consensus.cc:2781] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:41.701524 3531 raft_consensus.cc:686] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:41.702910 3531 consensus_queue.cc:237] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.718485 2156 catalog_manager.cc:5484] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:15:41.721937 2158 catalog_manager.cc:5484] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:41.733011 3626 ts_tablet_manager.cc:1398] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.043s user 0.010s sys 0.020s
I20240503 14:15:41.734700 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af99824d1853471b867e4ff1f01e9367" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:41.734989 2691 raft_consensus.cc:2379] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 685c6181561b4056abdecfd6d8bdb3b2 in current term 1: Already voted for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in this term.
I20240503 14:15:41.735491 2365 leader_election.cc:302] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2; no voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829
I20240503 14:15:41.735950 3591 raft_consensus.cc:2726] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:41.815304 3611 raft_consensus.cc:482] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:41.815541 3611 raft_consensus.cc:504] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.816125 3611 leader_election.cc:288] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:41.816709 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f139d77b0f0c47809e6408b2771adb73" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:41.816723 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f139d77b0f0c47809e6408b2771adb73" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:41.816946 2301 raft_consensus.cc:2454] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:41.816942 2562 raft_consensus.cc:2454] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:41.817369 2626 leader_election.cc:302] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:41.817632 3611 raft_consensus.cc:2781] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:41.817745 3611 raft_consensus.cc:482] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:41.817838 3611 raft_consensus.cc:3037] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.819794 3611 raft_consensus.cc:504] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.820400 3611 leader_election.cc:288] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:41.820886 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f139d77b0f0c47809e6408b2771adb73" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:41.820991 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f139d77b0f0c47809e6408b2771adb73" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:41.821131 2562 raft_consensus.cc:3037] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.821230 2301 raft_consensus.cc:3037] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:41.824021 2562 raft_consensus.cc:2454] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:41.824040 2301 raft_consensus.cc:2454] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:41.824465 2625 leader_election.cc:302] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:41.824748 3611 raft_consensus.cc:2781] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:41.824908 3611 raft_consensus.cc:686] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:41.825177 3611 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:41.827819 2158 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:15:42.131294 3641 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.136056 3531 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:42.200963 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:42.201371 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:42.209064 3646 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:42.219290 3591 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.227061 3591 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 14:15:42.405763 3611 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:42.414675 3611 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 226.77 ms
time per row: 0.113385 ms
I20240503 14:15:42.594080 2154 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:42.603111 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 6325a400641b4d25b95592abb4af7d2d (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.603157 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 6325a400641b4d25b95592abb4af7d2d (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.603404 2670 tablet_service.cc:1467] Processing CreateTablet for tablet 6325a400641b4d25b95592abb4af7d2d (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.603631 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6325a400641b4d25b95592abb4af7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.603636 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6325a400641b4d25b95592abb4af7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.603813 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6325a400641b4d25b95592abb4af7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.606384 2281 tablet_service.cc:1467] Processing CreateTablet for tablet b359fb8d9afc4c9c856897a722100304 (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.606804 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b359fb8d9afc4c9c856897a722100304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.608098 2410 tablet_service.cc:1467] Processing CreateTablet for tablet dc88c5dc94844f1c8400c31ad1165a32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.608505 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc88c5dc94844f1c8400c31ad1165a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.608578 2671 tablet_service.cc:1467] Processing CreateTablet for tablet dc88c5dc94844f1c8400c31ad1165a32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.608934 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc88c5dc94844f1c8400c31ad1165a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.613029 2411 tablet_service.cc:1467] Processing CreateTablet for tablet b359fb8d9afc4c9c856897a722100304 (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.613415 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b359fb8d9afc4c9c856897a722100304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.614097 2541 tablet_service.cc:1467] Processing CreateTablet for tablet dc88c5dc94844f1c8400c31ad1165a32 (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.614532 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc88c5dc94844f1c8400c31ad1165a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.615473 3713 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:42.617316 3713 tablet_bootstrap.cc:654] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.619556 2540 tablet_service.cc:1467] Processing CreateTablet for tablet b359fb8d9afc4c9c856897a722100304 (DEFAULT_TABLE table=rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:42.619951 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b359fb8d9afc4c9c856897a722100304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:42.629326 3716 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:42.633612 3716 tablet_bootstrap.cc:654] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.635936 3713 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:42.636090 3713 ts_tablet_manager.cc:1367] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20240503 14:15:42.637933 3713 raft_consensus.cc:348] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.638137 3713 raft_consensus.cc:374] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.638218 3713 raft_consensus.cc:729] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.638440 3713 consensus_queue.cc:260] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.639008 3713 ts_tablet_manager.cc:1398] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:15:42.639356 3713 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:42.640077 3716 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:42.640269 3716 ts_tablet_manager.cc:1367] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:15:42.641141 3713 tablet_bootstrap.cc:654] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.641188 3716 raft_consensus.cc:348] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.641542 3716 raft_consensus.cc:374] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.641685 3716 raft_consensus.cc:729] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.642022 3716 consensus_queue.cc:260] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.642760 3716 ts_tablet_manager.cc:1398] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.643076 3716 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:42.644302 3717 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:42.644726 3716 tablet_bootstrap.cc:654] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.646703 3717 tablet_bootstrap.cc:654] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.648586 3716 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:42.648754 3716 ts_tablet_manager.cc:1367] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:15:42.649135 3714 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:42.649488 3716 raft_consensus.cc:348] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.649655 3716 raft_consensus.cc:374] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.649729 3716 raft_consensus.cc:729] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.649770 3713 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:42.649924 3713 ts_tablet_manager.cc:1367] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:15:42.650282 3716 consensus_queue.cc:260] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.650682 3713 raft_consensus.cc:348] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.650825 3716 ts_tablet_manager.cc:1398] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.650928 3714 tablet_bootstrap.cc:654] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.650928 3713 raft_consensus.cc:374] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.650997 3713 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.651274 3713 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.652578 3713 ts_tablet_manager.cc:1398] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:42.652926 3713 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:42.654533 3713 tablet_bootstrap.cc:654] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.654646 3714 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:42.654847 3714 ts_tablet_manager.cc:1367] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:15:42.656860 3714 raft_consensus.cc:348] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.657276 3714 raft_consensus.cc:374] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.657399 3714 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.657697 3714 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.658159 3717 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:42.658282 3713 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:42.658411 3713 ts_tablet_manager.cc:1367] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:15:42.658419 3717 ts_tablet_manager.cc:1367] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 14:15:42.658406 3714 ts_tablet_manager.cc:1398] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:15:42.659250 3717 raft_consensus.cc:348] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.659494 3717 raft_consensus.cc:374] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.659613 3717 raft_consensus.cc:729] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.659847 3717 consensus_queue.cc:260] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.660537 3717 ts_tablet_manager.cc:1398] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.660804 3717 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:42.660909 3713 raft_consensus.cc:348] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.661085 3713 raft_consensus.cc:374] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.661173 3713 raft_consensus.cc:729] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.661439 3713 consensus_queue.cc:260] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.662070 3713 ts_tablet_manager.cc:1398] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:15:42.662329 3717 tablet_bootstrap.cc:654] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.664850 3717 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:42.665002 3717 ts_tablet_manager.cc:1367] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:42.665836 3717 raft_consensus.cc:348] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.666038 3717 raft_consensus.cc:374] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.666118 3717 raft_consensus.cc:729] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.666354 3717 consensus_queue.cc:260] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.666961 3717 ts_tablet_manager.cc:1398] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:42.667265 3717 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:42.668769 3717 tablet_bootstrap.cc:654] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:42.672487 3717 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:42.672693 3717 ts_tablet_manager.cc:1367] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240503 14:15:42.673346 3717 raft_consensus.cc:348] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.673499 3717 raft_consensus.cc:374] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:42.673595 3717 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:42.673794 3717 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.674216 3717 ts_tablet_manager.cc:1398] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:42.903087 3662 raft_consensus.cc:482] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.903290 3662 raft_consensus.cc:504] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.903873 3662 leader_election.cc:288] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:42.904441 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc88c5dc94844f1c8400c31ad1165a32" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:42.904453 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc88c5dc94844f1c8400c31ad1165a32" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:42.904685 2562 raft_consensus.cc:2454] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:42.904682 2691 raft_consensus.cc:2454] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:42.905094 2365 leader_election.cc:302] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:42.905319 3662 raft_consensus.cc:2781] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:42.905432 3662 raft_consensus.cc:482] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:42.905534 3662 raft_consensus.cc:3037] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.907366 3662 raft_consensus.cc:504] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.907837 3662 leader_election.cc:288] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:42.908268 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc88c5dc94844f1c8400c31ad1165a32" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:42.908298 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc88c5dc94844f1c8400c31ad1165a32" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:42.908457 2691 raft_consensus.cc:3037] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.908452 2562 raft_consensus.cc:3037] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.910274 2562 raft_consensus.cc:2454] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:42.910482 2691 raft_consensus.cc:2454] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:42.910646 2365 leader_election.cc:302] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:42.910962 3662 raft_consensus.cc:2781] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:42.911106 3662 raft_consensus.cc:686] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:42.911412 3662 consensus_queue.cc:237] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.913938 2154 catalog_manager.cc:5484] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:42.940222 3656 raft_consensus.cc:482] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.940438 3656 raft_consensus.cc:504] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.941973 3656 leader_election.cc:288] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:42.951048 3662 raft_consensus.cc:482] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:42.951422 3662 raft_consensus.cc:504] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.952507 3662 leader_election.cc:288] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:42.953641 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:42.953897 2562 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:42.954286 2365 leader_election.cc:302] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:42.954613 3662 raft_consensus.cc:2781] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:42.954727 3662 raft_consensus.cc:482] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:42.954843 3662 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.956024 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:42.956269 2301 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:42.958189 3662 raft_consensus.cc:504] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.959295 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:42.959529 2562 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.959754 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:42.959981 2301 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.960743 3662 leader_election.cc:288] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:42.960873 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6325a400641b4d25b95592abb4af7d2d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:42.961104 2431 raft_consensus.cc:2454] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:42.961496 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6325a400641b4d25b95592abb4af7d2d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:42.961602 2496 leader_election.cc:302] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:42.961736 2691 raft_consensus.cc:2454] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:42.962136 3656 raft_consensus.cc:2781] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:42.962261 3656 raft_consensus.cc:482] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:42.962357 3656 raft_consensus.cc:3037] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.962365 2562 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:42.962869 2301 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:42.963382 2365 leader_election.cc:302] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:42.963712 3662 raft_consensus.cc:2781] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:42.963913 3662 raft_consensus.cc:686] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:42.964205 3662 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:42.964934 3656 raft_consensus.cc:504] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.965503 3656 leader_election.cc:288] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:42.965984 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6325a400641b4d25b95592abb4af7d2d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:42.966166 2431 raft_consensus.cc:3037] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.966591 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6325a400641b4d25b95592abb4af7d2d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:42.966809 2691 raft_consensus.cc:3037] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:42.967820 2154 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:42.968217 2431 raft_consensus.cc:2454] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:42.968602 2496 leader_election.cc:302] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:42.968919 3656 raft_consensus.cc:2781] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:42.969475 2691 raft_consensus.cc:2454] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:42.969741 3656 raft_consensus.cc:686] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:15:42.970247 3656 consensus_queue.cc:237] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:42.973673 2154 catalog_manager.cc:5484] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
W20240503 14:15:43.357259 3734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:43.357615 3734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:43.365283 3734 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 14:15:43.419922 3662 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:43.420045 3696 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:43.453668 3696 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:43.465179 3696 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:43.490278 3732 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:43.515529 3656 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 280.695 ms
time per row: 0.140347 ms
I20240503 14:15:43.792138 2160 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:43.801026 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 78edc267e9f44d82b7dbb663628359fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.801530 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edc267e9f44d82b7dbb663628359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.801535 2280 tablet_service.cc:1467] Processing CreateTablet for tablet abdff0b719a342cfbde746751bd2a89d (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.801957 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdff0b719a342cfbde746751bd2a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.802156 2279 tablet_service.cc:1467] Processing CreateTablet for tablet b112f8fb89c7404b87a4c6e3d72fdcfd (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.802577 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b112f8fb89c7404b87a4c6e3d72fdcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.802881 2540 tablet_service.cc:1467] Processing CreateTablet for tablet abdff0b719a342cfbde746751bd2a89d (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.803300 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdff0b719a342cfbde746751bd2a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.807358 2670 tablet_service.cc:1467] Processing CreateTablet for tablet 78edc267e9f44d82b7dbb663628359fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.807590 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 78edc267e9f44d82b7dbb663628359fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.807804 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edc267e9f44d82b7dbb663628359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.807979 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edc267e9f44d82b7dbb663628359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.811726 2671 tablet_service.cc:1467] Processing CreateTablet for tablet abdff0b719a342cfbde746751bd2a89d (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.812250 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdff0b719a342cfbde746751bd2a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.812184 2411 tablet_service.cc:1467] Processing CreateTablet for tablet b112f8fb89c7404b87a4c6e3d72fdcfd (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.812711 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b112f8fb89c7404b87a4c6e3d72fdcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.815671 2672 tablet_service.cc:1467] Processing CreateTablet for tablet b112f8fb89c7404b87a4c6e3d72fdcfd (DEFAULT_TABLE table=rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:43.816155 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b112f8fb89c7404b87a4c6e3d72fdcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:43.818408 3796 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:43.821404 3798 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:43.823241 3798 tablet_bootstrap.cc:654] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.824659 3797 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:43.825322 3796 tablet_bootstrap.cc:654] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.826457 3797 tablet_bootstrap.cc:654] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.827430 3798 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:43.827630 3798 ts_tablet_manager.cc:1367] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:15:43.828485 3798 raft_consensus.cc:348] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:43.828827 3798 raft_consensus.cc:374] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.828955 3798 raft_consensus.cc:729] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.829270 3798 consensus_queue.cc:260] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:43.833472 3796 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:43.833745 3796 ts_tablet_manager.cc:1367] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240503 14:15:43.834502 3797 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:43.834502 3801 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:43.834647 3797 ts_tablet_manager.cc:1367] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240503 14:15:43.834721 3796 raft_consensus.cc:348] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.834939 3796 raft_consensus.cc:374] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.835028 3796 raft_consensus.cc:729] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.835278 3796 consensus_queue.cc:260] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.835383 3797 raft_consensus.cc:348] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.835608 3797 raft_consensus.cc:374] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.835747 3797 raft_consensus.cc:729] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.836012 3796 ts_tablet_manager.cc:1398] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:43.836024 3797 consensus_queue.cc:260] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.836627 3801 tablet_bootstrap.cc:654] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.837091 3798 ts_tablet_manager.cc:1398] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:15:43.837409 3797 ts_tablet_manager.cc:1398] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:43.837435 3796 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:43.839049 3801 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:43.839113 3796 tablet_bootstrap.cc:654] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.839507 3801 ts_tablet_manager.cc:1367] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:43.840493 3798 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:43.840397 3801 raft_consensus.cc:348] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.840730 3801 raft_consensus.cc:374] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.840868 3801 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.841344 3801 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.842115 3801 ts_tablet_manager.cc:1398] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:43.842339 3798 tablet_bootstrap.cc:654] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.842474 3801 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:43.844622 3801 tablet_bootstrap.cc:654] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.847687 3796 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:43.847878 3796 ts_tablet_manager.cc:1367] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240503 14:15:43.848452 3801 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:43.848635 3801 ts_tablet_manager.cc:1367] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 14:15:43.848708 3796 raft_consensus.cc:348] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.848970 3796 raft_consensus.cc:374] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.849049 3796 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.849315 3796 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.849427 3801 raft_consensus.cc:348] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.849630 3801 raft_consensus.cc:374] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.849716 3801 raft_consensus.cc:729] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.849949 3796 ts_tablet_manager.cc:1398] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:43.850049 3801 consensus_queue.cc:260] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.850390 3796 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:43.850769 3801 ts_tablet_manager.cc:1398] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:15:43.851068 3801 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:43.851172 3798 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:43.851450 3798 ts_tablet_manager.cc:1367] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:15:43.852528 3798 raft_consensus.cc:348] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.852852 3798 raft_consensus.cc:374] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.852928 3801 tablet_bootstrap.cc:654] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.852988 3798 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.853327 3798 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.855763 3796 tablet_bootstrap.cc:654] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:43.856808 3798 ts_tablet_manager.cc:1398] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:15:43.859375 3796 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:43.859594 3801 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:43.859603 3796 ts_tablet_manager.cc:1367] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:15:43.859752 3801 ts_tablet_manager.cc:1367] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:15:43.860394 3796 raft_consensus.cc:348] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:43.860507 3801 raft_consensus.cc:348] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:43.860735 3801 raft_consensus.cc:374] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.860738 3796 raft_consensus.cc:374] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:43.860850 3801 raft_consensus.cc:729] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.860850 3796 raft_consensus.cc:729] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:43.861088 3796 consensus_queue.cc:260] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:43.861093 3801 consensus_queue.cc:260] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:43.861672 3801 ts_tablet_manager.cc:1398] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:43.861835 3796 ts_tablet_manager.cc:1398] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:43.962141 3611 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:43.962355 3611 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.962981 3611 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:43.963671 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:43.963672 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:43.963925 2562 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:43.963925 2300 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:43.964387 2626 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:43.964756 3611 raft_consensus.cc:2781] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:43.964871 3611 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:43.964996 3611 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:43.966960 3611 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.967576 3611 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:43.968032 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:43.968040 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:43.968199 2562 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:43.968214 2300 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:43.970196 2562 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:43.970232 2300 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:43.970598 2625 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:43.970861 3611 raft_consensus.cc:2781] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:43.971263 3611 raft_consensus.cc:686] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:43.971545 3611 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:43.974212 2158 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:43.986398 3611 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:43.986577 3611 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.987319 3611 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:43.987916 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:43.987975 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:43.988199 2431 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:43.988198 2300 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:43.988610 2626 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:43.988893 3611 raft_consensus.cc:2781] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:43.989023 3611 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:43.989109 3611 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:43.991429 3611 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.991942 3611 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:43.992384 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:43.992393 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:43.992630 2431 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:43.992645 2300 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:43.994544 2300 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:43.994578 2431 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:43.994920 2626 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:43.995240 3611 raft_consensus.cc:2781] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:43.995409 3611 raft_consensus.cc:686] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:43.995704 3611 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:43.998615 2158 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:44.059288 3809 raft_consensus.cc:482] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:44.059471 3809 raft_consensus.cc:504] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.060251 3809 leader_election.cc:288] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:44.060778 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:44.060822 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:44.060982 2300 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:44.061079 2431 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:44.061523 2626 leader_election.cc:302] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:44.061796 3809 raft_consensus.cc:2781] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:44.061906 3809 raft_consensus.cc:482] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:44.062086 3809 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.064879 3809 raft_consensus.cc:504] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.065428 3809 leader_election.cc:288] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:44.065855 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:44.065919 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:44.066020 2300 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.066121 2431 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.067909 2300 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:44.067987 2431 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:44.068312 2626 leader_election.cc:302] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:44.068629 3809 raft_consensus.cc:2781] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:44.068784 3809 raft_consensus.cc:686] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:44.069063 3809 consensus_queue.cc:237] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.071759 2158 catalog_manager.cc:5484] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:15:44.382835 3809 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.389597 3812 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:44.481655 3814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:44.482051 3814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:44.490028 3814 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:44.490332 3812 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.495421 3812 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240503 14:15:44.546211 3812 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:44.551474 3812 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 245.489 ms
time per row: 0.122744 ms
I20240503 14:15:44.886144 2160 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:44.895746 2540 tablet_service.cc:1467] Processing CreateTablet for tablet a9a474791f9b4585aa0114162705c96f (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.895927 2279 tablet_service.cc:1467] Processing CreateTablet for tablet a9a474791f9b4585aa0114162705c96f (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.896246 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a474791f9b4585aa0114162705c96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.896396 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a474791f9b4585aa0114162705c96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.898810 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 832f07a6b12740d08fa67d653b3818bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.898828 2542 tablet_service.cc:1467] Processing CreateTablet for tablet ff59573e2c8f472585b96ca0a0fd75b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.898924 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 832f07a6b12740d08fa67d653b3818bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.899246 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832f07a6b12740d08fa67d653b3818bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.899289 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59573e2c8f472585b96ca0a0fd75b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.899590 2670 tablet_service.cc:1467] Processing CreateTablet for tablet ff59573e2c8f472585b96ca0a0fd75b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.899587 2411 tablet_service.cc:1467] Processing CreateTablet for tablet a9a474791f9b4585aa0114162705c96f (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.899964 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59573e2c8f472585b96ca0a0fd75b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.899998 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a474791f9b4585aa0114162705c96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.903532 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 832f07a6b12740d08fa67d653b3818bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.903892 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832f07a6b12740d08fa67d653b3818bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.906101 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832f07a6b12740d08fa67d653b3818bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.908105 2281 tablet_service.cc:1467] Processing CreateTablet for tablet ff59573e2c8f472585b96ca0a0fd75b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:44.908599 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59573e2c8f472585b96ca0a0fd75b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:44.913964 3873 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:44.915807 3873 tablet_bootstrap.cc:654] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.928244 3876 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:44.929240 3872 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:44.929486 3873 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:44.929692 3873 ts_tablet_manager.cc:1367] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240503 14:15:44.930559 3873 raft_consensus.cc:348] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:44.930835 3873 raft_consensus.cc:374] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.930976 3873 raft_consensus.cc:729] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.931107 3876 tablet_bootstrap.cc:654] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.931281 3872 tablet_bootstrap.cc:654] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.931308 3873 consensus_queue.cc:260] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:44.933061 3873 ts_tablet_manager.cc:1398] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:44.933459 3873 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:44.936851 3873 tablet_bootstrap.cc:654] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.940459 3876 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:44.940618 3876 ts_tablet_manager.cc:1367] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 14:15:44.940968 3875 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:44.941563 3876 raft_consensus.cc:348] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.941838 3876 raft_consensus.cc:374] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.941990 3876 raft_consensus.cc:729] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.942256 3876 consensus_queue.cc:260] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.942898 3876 ts_tablet_manager.cc:1398] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:44.943208 3876 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:44.943573 3875 tablet_bootstrap.cc:654] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.944840 3876 tablet_bootstrap.cc:654] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.947253 3872 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:44.947458 3872 ts_tablet_manager.cc:1367] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240503 14:15:44.948383 3872 raft_consensus.cc:348] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.948838 3872 raft_consensus.cc:374] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.948982 3872 raft_consensus.cc:729] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.949074 3876 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:44.949229 3876 ts_tablet_manager.cc:1367] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:15:44.949368 3872 consensus_queue.cc:260] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.949816 3873 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:44.949852 3875 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:44.949987 3875 ts_tablet_manager.cc:1367] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:15:44.950088 3873 ts_tablet_manager.cc:1367] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240503 14:15:44.949983 3876 raft_consensus.cc:348] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:44.950188 3876 raft_consensus.cc:374] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.950273 3876 raft_consensus.cc:729] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.950562 3876 consensus_queue.cc:260] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:44.951066 3872 ts_tablet_manager.cc:1398] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:15:44.951223 3875 raft_consensus.cc:348] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.951270 3873 raft_consensus.cc:348] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.951448 3875 raft_consensus.cc:374] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.951491 3872 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:44.951583 3875 raft_consensus.cc:729] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.951615 3873 raft_consensus.cc:374] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.951694 3873 raft_consensus.cc:729] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.951853 3875 consensus_queue.cc:260] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.951964 3873 consensus_queue.cc:260] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.952416 3876 ts_tablet_manager.cc:1398] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:44.952593 3873 ts_tablet_manager.cc:1398] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:44.953212 3875 ts_tablet_manager.cc:1398] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:44.953608 3875 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:44.955230 3875 tablet_bootstrap.cc:654] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.955683 3872 tablet_bootstrap.cc:654] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.959429 3872 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:44.959596 3872 ts_tablet_manager.cc:1367] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:15:44.959844 3875 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:44.960160 3875 ts_tablet_manager.cc:1367] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 14:15:44.960366 3872 raft_consensus.cc:348] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.960605 3872 raft_consensus.cc:374] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.960695 3872 raft_consensus.cc:729] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.960939 3875 raft_consensus.cc:348] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.960942 3872 consensus_queue.cc:260] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:44.961151 3875 raft_consensus.cc:374] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.961232 3875 raft_consensus.cc:729] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.961427 3872 ts_tablet_manager.cc:1398] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:44.961477 3875 consensus_queue.cc:260] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.961782 3872 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:44.962014 3875 ts_tablet_manager.cc:1398] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:44.963223 3872 tablet_bootstrap.cc:654] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:44.965900 3872 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:44.966037 3872 ts_tablet_manager.cc:1367] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:44.966711 3872 raft_consensus.cc:348] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:44.966859 3872 raft_consensus.cc:374] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:44.966955 3872 raft_consensus.cc:729] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:44.967151 3872 consensus_queue.cc:260] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:44.967659 3872 ts_tablet_manager.cc:1398] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:44.988101 3861 raft_consensus.cc:482] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:44.988260 3861 raft_consensus.cc:504] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.989006 3861 leader_election.cc:288] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:44.989459 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59573e2c8f472585b96ca0a0fd75b1" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:44.989511 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59573e2c8f472585b96ca0a0fd75b1" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:44.989706 2560 raft_consensus.cc:2454] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:44.989763 2300 raft_consensus.cc:2454] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:44.990190 2626 leader_election.cc:302] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:44.990453 3861 raft_consensus.cc:2781] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:44.990585 3861 raft_consensus.cc:482] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:44.990656 3861 raft_consensus.cc:3037] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.992617 3861 raft_consensus.cc:504] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:44.993125 3861 leader_election.cc:288] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:44.993690 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59573e2c8f472585b96ca0a0fd75b1" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:44.993765 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff59573e2c8f472585b96ca0a0fd75b1" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:44.993975 2560 raft_consensus.cc:3037] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.993983 2300 raft_consensus.cc:3037] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:44.996111 2560 raft_consensus.cc:2454] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:44.996196 2300 raft_consensus.cc:2454] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:44.996601 2626 leader_election.cc:302] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:44.996928 3861 raft_consensus.cc:2781] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:44.997066 3861 raft_consensus.cc:686] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:44.997586 3861 consensus_queue.cc:237] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:15:45.000435 2156 catalog_manager.cc:5484] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:15:45.092470 3847 raft_consensus.cc:482] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:45.092783 3861 raft_consensus.cc:482] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:45.092730 3847 raft_consensus.cc:504] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:45.092989 3861 raft_consensus.cc:504] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:45.093578 3847 leader_election.cc:288] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:45.094074 3861 leader_election.cc:288] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:45.094134 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832f07a6b12740d08fa67d653b3818bf" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:45.094179 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:45.094370 2431 raft_consensus.cc:2454] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:45.094415 2300 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:45.094439 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:45.094524 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832f07a6b12740d08fa67d653b3818bf" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:45.094710 2560 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:45.094760 2562 raft_consensus.cc:2454] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 0.
I20240503 14:15:45.094806 2366 leader_election.cc:302] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:45.094935 2626 leader_election.cc:302] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:45.095257 3847 raft_consensus.cc:2781] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:45.095284 3861 raft_consensus.cc:2781] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:45.095371 3847 raft_consensus.cc:482] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:45.095376 3861 raft_consensus.cc:482] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:45.095460 3847 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.095489 3861 raft_consensus.cc:3037] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.097391 3847 raft_consensus.cc:504] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:45.097426 3861 raft_consensus.cc:504] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:45.098004 3847 leader_election.cc:288] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:45.098163 3861 leader_election.cc:288] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:45.098508 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:45.098573 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832f07a6b12740d08fa67d653b3818bf" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:45.098711 2300 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.098795 2431 raft_consensus.cc:3037] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.098744 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:45.098970 2560 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.099524 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "832f07a6b12740d08fa67d653b3818bf" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:45.099738 2562 raft_consensus.cc:3037] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.101581 2431 raft_consensus.cc:2454] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:45.101610 2300 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:45.101931 2560 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:45.102025 2366 leader_election.cc:302] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:45.102283 2626 leader_election.cc:302] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:45.102334 3847 raft_consensus.cc:2781] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:45.102464 2562 raft_consensus.cc:2454] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 1.
I20240503 14:15:45.102581 3847 raft_consensus.cc:686] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:45.102869 3861 raft_consensus.cc:2781] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:45.103035 3861 raft_consensus.cc:686] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:15:45.102882 3847 consensus_queue.cc:237] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:15:45.103320 3861 consensus_queue.cc:237] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:45.107909 2156 catalog_manager.cc:5484] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:45.108126 2160 catalog_manager.cc:5484] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:15:45.418071 3861 consensus_queue.cc:1035] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.424644 3812 consensus_queue.cc:1035] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:45.513988 3889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:45.514382 3889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:45.522248 3889 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 14:15:45.570766 3861 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.575574 3812 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.577560 3762 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:45.589287 3847 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:45.807534 2626 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 50536.8us
INSERT report
rows total: 2000
time total: 299.54 ms
time per row: 0.14977 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:15:46.929880 2160 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:46.938635 2279 tablet_service.cc:1467] Processing CreateTablet for tablet aba6537cdb87400cb8b108b1fbfa6c0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.939127 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba6537cdb87400cb8b108b1fbfa6c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.939294 2542 tablet_service.cc:1467] Processing CreateTablet for tablet aba6537cdb87400cb8b108b1fbfa6c0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.939752 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba6537cdb87400cb8b108b1fbfa6c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.940420 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae190d0f4d843c990e6fdadcac677ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.940434 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae190d0f4d843c990e6fdadcac677ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.940816 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae190d0f4d843c990e6fdadcac677ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.940855 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae190d0f4d843c990e6fdadcac677ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.943885 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae190d0f4d843c990e6fdadcac677ed (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.944298 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae190d0f4d843c990e6fdadcac677ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.944202 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 059a4a35aa94412994bb431342ba0753 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.944192 2410 tablet_service.cc:1467] Processing CreateTablet for tablet aba6537cdb87400cb8b108b1fbfa6c0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.944574 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a4a35aa94412994bb431342ba0753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.944588 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba6537cdb87400cb8b108b1fbfa6c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.947326 2670 tablet_service.cc:1467] Processing CreateTablet for tablet 059a4a35aa94412994bb431342ba0753 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.947721 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a4a35aa94412994bb431342ba0753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.950868 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 059a4a35aa94412994bb431342ba0753 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:46.951745 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a4a35aa94412994bb431342ba0753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:46.964730 3962 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:46.965586 3963 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:46.966512 3962 tablet_bootstrap.cc:654] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.967832 3960 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:46.969210 3960 tablet_bootstrap.cc:654] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.969444 3963 tablet_bootstrap.cc:654] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.970855 3962 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:46.971006 3962 ts_tablet_manager.cc:1367] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:15:46.971562 3961 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:46.971796 3962 raft_consensus.cc:348] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.971999 3962 raft_consensus.cc:374] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.972115 3962 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.972355 3962 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.972863 3962 ts_tablet_manager.cc:1398] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:46.972987 3961 tablet_bootstrap.cc:654] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.973460 3962 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:46.974819 3962 tablet_bootstrap.cc:654] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.976313 3960 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:46.976562 3960 ts_tablet_manager.cc:1367] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:15:46.977380 3961 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:46.977638 3961 ts_tablet_manager.cc:1367] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:15:46.978531 3955 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:46.978616 3961 raft_consensus.cc:348] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.978675 3955 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.978821 3961 raft_consensus.cc:374] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.978906 3961 raft_consensus.cc:729] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.979095 3963 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:46.979261 3963 ts_tablet_manager.cc:1367] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20240503 14:15:46.979380 3955 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:46.979315 3961 consensus_queue.cc:260] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.979976 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:46.980054 3963 raft_consensus.cc:348] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:46.980265 3963 raft_consensus.cc:374] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.980377 3963 raft_consensus.cc:729] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.980600 3963 consensus_queue.cc:260] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:46.980779 3961 ts_tablet_manager.cc:1398] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:46.981091 3961 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:46.981141 3963 ts_tablet_manager.cc:1398] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:46.980607 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
W20240503 14:15:46.982919 2234 leader_election.cc:341] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:46.983259 2234 leader_election.cc:341] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:46.983373 2234 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829
I20240503 14:15:46.983790 3955 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:46.984028 3961 tablet_bootstrap.cc:654] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.985018 3963 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:46.984954 3960 raft_consensus.cc:348] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:46.985229 3960 raft_consensus.cc:374] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.985369 3960 raft_consensus.cc:729] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.985749 3960 consensus_queue.cc:260] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:46.986670 3960 ts_tablet_manager.cc:1398] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:15:46.986785 3963 tablet_bootstrap.cc:654] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.987223 3960 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:46.990285 3960 tablet_bootstrap.cc:654] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:46.995918 3962 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:46.996227 3962 ts_tablet_manager.cc:1367] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20240503 14:15:46.997262 3962 raft_consensus.cc:348] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.997648 3963 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:46.997711 3962 raft_consensus.cc:374] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.997879 3962 raft_consensus.cc:729] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.997946 3963 ts_tablet_manager.cc:1367] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20240503 14:15:46.998344 3962 consensus_queue.cc:260] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.998912 3963 raft_consensus.cc:348] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:46.999148 3962 ts_tablet_manager.cc:1398] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:46.999199 3963 raft_consensus.cc:374] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:46.999338 3963 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:46.999418 3960 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:46.999578 3962 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:46.999708 3963 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.001425 3960 ts_tablet_manager.cc:1367] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240503 14:15:47.001693 3962 tablet_bootstrap.cc:654] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:47.002647 3960 raft_consensus.cc:348] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.002933 3960 raft_consensus.cc:374] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.003043 3960 raft_consensus.cc:729] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.003377 3960 consensus_queue.cc:260] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.004051 3963 ts_tablet_manager.cc:1398] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:47.004376 3960 ts_tablet_manager.cc:1398] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:47.007686 3961 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:47.007964 3961 ts_tablet_manager.cc:1367] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.010s
I20240503 14:15:47.008862 3961 raft_consensus.cc:348] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.009140 3961 raft_consensus.cc:374] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.009246 3961 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.009317 3962 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:47.009461 3962 ts_tablet_manager.cc:1367] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240503 14:15:47.009564 3961 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.010205 3961 ts_tablet_manager.cc:1398] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:47.010183 3962 raft_consensus.cc:348] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:47.010401 3962 raft_consensus.cc:374] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:47.010496 3962 raft_consensus.cc:729] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:47.010712 3962 consensus_queue.cc:260] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:47.011273 3962 ts_tablet_manager.cc:1398] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:47.068598 3788 raft_consensus.cc:482] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.068775 3788 raft_consensus.cc:504] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.069463 3788 leader_election.cc:288] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:47.070612 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba6537cdb87400cb8b108b1fbfa6c0f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:47.070860 2432 raft_consensus.cc:2454] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:47.071334 2496 leader_election.cc:302] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:47.071619 3788 raft_consensus.cc:2781] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.071758 3788 raft_consensus.cc:482] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.071866 3788 raft_consensus.cc:3037] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.074304 3788 raft_consensus.cc:504] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.074949 3788 leader_election.cc:288] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:47.075479 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba6537cdb87400cb8b108b1fbfa6c0f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:47.075701 2432 raft_consensus.cc:3037] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.076653 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba6537cdb87400cb8b108b1fbfa6c0f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:47.076763 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aba6537cdb87400cb8b108b1fbfa6c0f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:47.076885 2301 raft_consensus.cc:2454] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:47.077904 2432 raft_consensus.cc:2454] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:47.078238 2496 leader_election.cc:302] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters: 7f22b4d5b1b5466dbbdd344f0721c05f
I20240503 14:15:47.078510 3788 raft_consensus.cc:2781] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:47.078876 3788 raft_consensus.cc:686] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:15:47.079129 3788 consensus_queue.cc:237] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.082010 2157 catalog_manager.cc:5484] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:47.114187 3788 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.114413 3788 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.115089 3788 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:47.115571 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:47.115662 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:47.115856 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:47.115890 2299 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:47.116281 2495 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:47.116622 3788 raft_consensus.cc:2781] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.116739 3788 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.116849 3788 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.118721 3788 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.119381 3788 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:47.119753 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:47.119963 2691 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.119901 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:47.120085 2301 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.122018 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:47.122017 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:47.122400 2495 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:47.122676 3788 raft_consensus.cc:2781] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:47.122871 3788 raft_consensus.cc:686] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:15:47.123140 3788 consensus_queue.cc:237] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:47.126427 2157 catalog_manager.cc:5484] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:47.137809 3955 raft_consensus.cc:482] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:47.137991 3955 raft_consensus.cc:504] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:47.138615 3955 leader_election.cc:288] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:47.139127 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae190d0f4d843c990e6fdadcac677ed" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:47.139122 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae190d0f4d843c990e6fdadcac677ed" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:47.139386 2432 raft_consensus.cc:2454] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:47.139372 2691 raft_consensus.cc:2454] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 0.
I20240503 14:15:47.139765 2235 leader_election.cc:302] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:47.140033 3955 raft_consensus.cc:2781] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:47.140166 3955 raft_consensus.cc:482] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:47.140249 3955 raft_consensus.cc:3037] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.142567 3955 raft_consensus.cc:504] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:47.143100 3955 leader_election.cc:288] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:47.143489 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae190d0f4d843c990e6fdadcac677ed" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:47.143585 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae190d0f4d843c990e6fdadcac677ed" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:47.143680 2432 raft_consensus.cc:3037] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.143766 2691 raft_consensus.cc:3037] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.145514 2432 raft_consensus.cc:2454] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:47.145586 2691 raft_consensus.cc:2454] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 1.
I20240503 14:15:47.145920 2235 leader_election.cc:302] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:15:47.146221 3955 raft_consensus.cc:2781] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:47.146373 3955 raft_consensus.cc:686] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Leader. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:15:47.146646 3955 consensus_queue.cc:237] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:47.149487 2157 catalog_manager.cc:5484] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1). New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:47.488466 2301 raft_consensus.cc:3037] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:47.491094 3788 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:47.496222 3788 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:47.573153 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:47.573509 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:15:47.577863 3788 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:47.582090 3788 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:47.583730 3979 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 14:15:47.620504 3952 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:47.626508 3955 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:47.747799 4005 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.20.99.62:33535, user_credentials={real_user=slave}} blocked reactor thread for 43954.3us
INSERT report
rows total: 2000
time total: 316.27 ms
time per row: 0.158135 ms
I20240503 14:15:48.058172 2155 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50520:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:48.066653 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 32927a23842b4725b9dfafbac47fe754 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.067067 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.066954 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 32927a23842b4725b9dfafbac47fe754 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.067397 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.068562 2280 tablet_service.cc:1467] Processing CreateTablet for tablet 57c4a133b5c646f78b07f37dc85904f6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.069006 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c4a133b5c646f78b07f37dc85904f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.068965 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 57c4a133b5c646f78b07f37dc85904f6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.069397 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c4a133b5c646f78b07f37dc85904f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.071591 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 5f631c7713d24f4caf7550bb05147615 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.071996 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f631c7713d24f4caf7550bb05147615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.075762 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 32927a23842b4725b9dfafbac47fe754 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.076165 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.078460 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 57c4a133b5c646f78b07f37dc85904f6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.078836 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c4a133b5c646f78b07f37dc85904f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.082487 2670 tablet_service.cc:1467] Processing CreateTablet for tablet 5f631c7713d24f4caf7550bb05147615 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.082857 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f631c7713d24f4caf7550bb05147615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.084182 4041 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:48.086220 4040 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:15:48.087118 4039 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:48.088013 4041 tablet_bootstrap.cc:654] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.088506 4040 tablet_bootstrap.cc:654] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.089414 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 5f631c7713d24f4caf7550bb05147615 (DEFAULT_TABLE table=rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:48.089874 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f631c7713d24f4caf7550bb05147615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:48.090787 4039 tablet_bootstrap.cc:654] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.096825 4041 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:48.097137 4041 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:15:48.098286 4041 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.098935 4041 raft_consensus.cc:374] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.099108 4041 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.099505 4041 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.100498 4041 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:48.100875 4041 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:15:48.102859 4041 tablet_bootstrap.cc:654] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.104380 4039 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:48.104635 4039 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240503 14:15:48.105122 4045 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:48.105764 4039 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.106149 4039 raft_consensus.cc:374] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.106309 4039 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.106846 4040 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f: No bootstrap required, opened a new log
I20240503 14:15:48.106714 4039 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.107084 4040 ts_tablet_manager.cc:1367] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20240503 14:15:48.107163 4045 tablet_bootstrap.cc:654] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.108011 4040 raft_consensus.cc:348] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.108268 4040 raft_consensus.cc:374] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.108417 4040 raft_consensus.cc:729] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.109109 4040 consensus_queue.cc:260] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.110994 4041 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829: No bootstrap required, opened a new log
I20240503 14:15:48.111215 4040 ts_tablet_manager.cc:1398] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:48.111236 4041 ts_tablet_manager.cc:1367] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.008s
I20240503 14:15:48.111809 4039 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:15:48.112011 4041 raft_consensus.cc:348] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.112203 4039 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:48.112344 4041 raft_consensus.cc:374] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.112504 4041 raft_consensus.cc:729] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.112808 4041 consensus_queue.cc:260] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.113493 4041 ts_tablet_manager.cc:1398] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:48.116520 4039 tablet_bootstrap.cc:654] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.118749 4039 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:48.118903 4039 ts_tablet_manager.cc:1367] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 14:15:48.119124 4045 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:48.119336 4045 ts_tablet_manager.cc:1367] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20240503 14:15:48.119701 4039 raft_consensus.cc:348] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.119879 4039 raft_consensus.cc:374] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.119958 4039 raft_consensus.cc:729] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.120103 4045 raft_consensus.cc:348] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.120230 4039 consensus_queue.cc:260] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.120359 4045 raft_consensus.cc:374] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.120458 4045 raft_consensus.cc:729] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.120751 4039 ts_tablet_manager.cc:1398] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:48.120697 4045 consensus_queue.cc:260] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.121176 4039 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:15:48.121246 4045 ts_tablet_manager.cc:1398] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:48.121598 4045 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:48.122893 4039 tablet_bootstrap.cc:654] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.123203 4045 tablet_bootstrap.cc:654] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.125037 4039 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: No bootstrap required, opened a new log
I20240503 14:15:48.125219 4039 ts_tablet_manager.cc:1367] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:15:48.126117 4039 raft_consensus.cc:348] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.126282 4039 raft_consensus.cc:374] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.126400 4039 raft_consensus.cc:729] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.126649 4039 consensus_queue.cc:260] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.127032 4045 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:48.127252 4039 ts_tablet_manager.cc:1398] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:15:48.127265 4045 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:15:48.128002 4045 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.128197 4045 raft_consensus.cc:374] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.128285 4045 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.128532 4045 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.129134 4045 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:48.129491 4045 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:15:48.131106 4045 tablet_bootstrap.cc:654] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:48.132891 4045 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: No bootstrap required, opened a new log
I20240503 14:15:48.133037 4045 ts_tablet_manager.cc:1367] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:48.133772 4045 raft_consensus.cc:348] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.133991 4045 raft_consensus.cc:374] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:48.134100 4045 raft_consensus.cc:729] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:15:48.134374 4045 consensus_queue.cc:260] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.134865 4045 ts_tablet_manager.cc:1398] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:48.157163 4033 raft_consensus.cc:482] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:48.157398 4033 raft_consensus.cc:504] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.158186 4033 leader_election.cc:288] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:48.158738 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:48.158948 2691 raft_consensus.cc:2454] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:48.159238 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:48.159426 2495 leader_election.cc:302] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:15:48.159478 2431 raft_consensus.cc:2454] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:48.159726 4033 raft_consensus.cc:2781] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:48.159878 4033 raft_consensus.cc:482] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:48.159983 4033 raft_consensus.cc:3037] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.162405 4033 raft_consensus.cc:504] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.163018 4033 leader_election.cc:288] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:48.163390 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:48.163605 2431 raft_consensus.cc:3037] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.163529 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:48.163791 2691 raft_consensus.cc:3037] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.165654 2431 raft_consensus.cc:2454] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:48.165937 2691 raft_consensus.cc:2454] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:48.166030 2496 leader_election.cc:302] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:48.166277 4033 raft_consensus.cc:2781] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:48.166430 4033 raft_consensus.cc:686] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:15:48.166749 4033 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.169672 2157 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:48.216673 4033 raft_consensus.cc:482] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:48.216915 4033 raft_consensus.cc:504] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.217520 4033 leader_election.cc:288] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:48.218036 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c4a133b5c646f78b07f37dc85904f6" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:15:48.218065 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c4a133b5c646f78b07f37dc85904f6" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:15:48.218254 2301 raft_consensus.cc:2454] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:48.218276 2431 raft_consensus.cc:2454] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 0.
I20240503 14:15:48.218645 2496 leader_election.cc:302] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:48.218925 4033 raft_consensus.cc:2781] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:48.219003 4033 raft_consensus.cc:482] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:48.219101 4033 raft_consensus.cc:3037] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.221040 4033 raft_consensus.cc:504] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.221534 4033 leader_election.cc:288] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:48.222018 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c4a133b5c646f78b07f37dc85904f6" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:15:48.222087 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57c4a133b5c646f78b07f37dc85904f6" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:15:48.222223 2301 raft_consensus.cc:3037] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.222358 2431 raft_consensus.cc:3037] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.224251 2431 raft_consensus.cc:2454] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:48.224256 2301 raft_consensus.cc:2454] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 1.
I20240503 14:15:48.224639 2496 leader_election.cc:302] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:48.224961 4033 raft_consensus.cc:2781] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:48.225142 4033 raft_consensus.cc:686] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:15:48.225437 4033 consensus_queue.cc:237] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:15:48.228286 2157 catalog_manager.cc:5484] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:15:48.230137 4015 raft_consensus.cc:482] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:48.230288 4015 raft_consensus.cc:504] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.230934 4015 leader_election.cc:288] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:48.231369 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f631c7713d24f4caf7550bb05147615" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:15:48.231493 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f631c7713d24f4caf7550bb05147615" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:15:48.231563 2561 raft_consensus.cc:2454] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:48.231721 2691 raft_consensus.cc:2454] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 0.
I20240503 14:15:48.231952 2365 leader_election.cc:302] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:48.232214 4015 raft_consensus.cc:2781] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:48.232324 4015 raft_consensus.cc:482] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:48.232414 4015 raft_consensus.cc:3037] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.234604 4015 raft_consensus.cc:504] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.235497 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f631c7713d24f4caf7550bb05147615" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:15:48.235083 4015 leader_election.cc:288] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:48.235688 2561 raft_consensus.cc:3037] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.235661 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f631c7713d24f4caf7550bb05147615" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:15:48.235848 2691 raft_consensus.cc:3037] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:48.238539 2691 raft_consensus.cc:2454] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:48.238538 2561 raft_consensus.cc:2454] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 1.
I20240503 14:15:48.239042 2365 leader_election.cc:302] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:15:48.239329 4015 raft_consensus.cc:2781] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:48.239477 4015 raft_consensus.cc:686] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:15:48.239740 4015 consensus_queue.cc:237] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:15:48.242511 2157 catalog_manager.cc:5484] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:15:48.621548 4033 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.625782 4015 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.647401 4013 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.656229 3975 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:48.661048 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:48.661713 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:48.677528 4055 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 14:15:48.721876 4033 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:48.726871 3975 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 254.277 ms
time per row: 0.127139 ms
I20240503 14:15:49.100351 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.5:41737
--local_ip_for_outbound_sockets=127.20.99.5
--tserver_master_addrs=127.20.99.62:33535
--webserver_port=41715
--webserver_interface=127.20.99.5
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:15:49.467216 4121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:49.467504 4121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:49.467643 4121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:49.475772 4121 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:49.475915 4121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:49.476084 4121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.5
I20240503 14:15:49.485675 4121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.5:41737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.99.5
--webserver_port=41715
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.5
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:49.487159 4121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:49.489188 4121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:49.503695 4126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:49.504384 4127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:49.505089 4129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:49.505931 4121 server_base.cc:1034] running on GCE node
I20240503 14:15:49.506647 4121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:49.507905 4121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:49.509174 4121 hybrid_clock.cc:648] HybridClock initialized: now 1714745749509075 us; error 97 us; skew 500 ppm
I20240503 14:15:49.509475 4121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:49.511749 4121 webserver.cc:460] Webserver started at http://127.20.99.5:41715/ using document root <none> and password file <none>
I20240503 14:15:49.512468 4121 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:49.512573 4121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:49.518106 4121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 14:15:49.520797 4135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.522082 4121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:15:49.522243 4121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7146de39e348442da4a363e95fbe0789"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:49.523028 4121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:49.536934 4121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:49.537686 4121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:49.537971 4121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:49.538857 4121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:49.540335 4121 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:49.540414 4121 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.540556 4121 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:49.540611 4121 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.575568 4121 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.5:41737
I20240503 14:15:49.575570 4247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.5:41737 every 8 connection(s)
I20240503 14:15:49.577152 4121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:15:49.580624 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 4121
I20240503 14:15:49.581386 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.6:42763
--local_ip_for_outbound_sockets=127.20.99.6
--tserver_master_addrs=127.20.99.62:33535
--webserver_port=36511
--webserver_interface=127.20.99.6
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:49.593020 4248 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:49.593359 4248 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:49.594308 4248 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:49.596822 2151 ts_manager.cc:194] Re-registered known tserver with Master: 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:15:49.598156 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.5:60369
W20240503 14:15:49.966888 4253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:49.967222 4253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:49.967361 4253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:49.975806 4253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:49.975968 4253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:49.976116 4253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.6
I20240503 14:15:49.986842 4253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.6:42763
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.99.6
--webserver_port=36511
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.6
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:49.988274 4253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:49.990010 4253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:50.002825 4258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:50.003465 4261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:50.004161 4259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:50.004966 4253 server_base.cc:1034] running on GCE node
I20240503 14:15:50.005584 4253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:50.006779 4253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:50.010115 4253 hybrid_clock.cc:648] HybridClock initialized: now 1714745750010011 us; error 90 us; skew 500 ppm
I20240503 14:15:50.010453 4253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:50.012777 4253 webserver.cc:460] Webserver started at http://127.20.99.6:36511/ using document root <none> and password file <none>
I20240503 14:15:50.013533 4253 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:50.013703 4253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:50.019106 4253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:15:50.021857 4267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:50.023196 4253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:50.023380 4253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "285488446ac445c5a5dfe654bfaf9835"
format_stamp: "Formatted at 2024-05-03 14:15:31 on dist-test-slave-p0gh"
I20240503 14:15:50.024317 4253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:50.044248 4253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:50.045053 4253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:50.045305 4253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:50.046236 4253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:50.047784 4253 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:50.047878 4253 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:50.047974 4253 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:50.048018 4253 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:50.089018 4253 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.6:42763
I20240503 14:15:50.089061 4379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.6:42763 every 8 connection(s)
I20240503 14:15:50.090418 4253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:15:50.098119 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 4253
I20240503 14:15:50.098884 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.7:44387
--local_ip_for_outbound_sockets=127.20.99.7
--tserver_master_addrs=127.20.99.62:33535
--webserver_port=39841
--webserver_interface=127.20.99.7
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:50.104089 4380 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:50.104400 4380 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:50.105147 4380 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:50.106573 2151 ts_manager.cc:194] Re-registered known tserver with Master: 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:15:50.107729 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.6:41467
W20240503 14:15:50.467605 4384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:50.467911 4384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:50.468008 4384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:50.476096 4384 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:50.476224 4384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:50.476342 4384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.7
I20240503 14:15:50.485471 4384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.7:44387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.99.7
--webserver_port=39841
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.7
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:50.486838 4384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:50.488965 4384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:50.502565 4393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:50.503445 4395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:50.504130 4392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:50.504963 4384 server_base.cc:1034] running on GCE node
I20240503 14:15:50.505554 4384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:50.506479 4384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:50.507665 4384 hybrid_clock.cc:648] HybridClock initialized: now 1714745750507608 us; error 45 us; skew 500 ppm
I20240503 14:15:50.507973 4384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:50.510412 4384 webserver.cc:460] Webserver started at http://127.20.99.7:39841/ using document root <none> and password file <none>
I20240503 14:15:50.511116 4384 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:50.511272 4384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:50.516613 4384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:15:50.519541 4401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:50.520866 4384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:15:50.521018 4384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d4cbc83202c8473d8c897582d30e5989"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:50.521787 4384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:50.543345 4384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:50.544273 4384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:50.544617 4384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:50.545601 4384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:50.547026 4384 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:50.547104 4384 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:50.547256 4384 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:50.547313 4384 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:50.588537 4384 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.7:44387
I20240503 14:15:50.588555 4513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.7:44387 every 8 connection(s)
I20240503 14:15:50.589973 4384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:15:50.595340 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 4384
I20240503 14:15:50.596060 20876 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
/tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.99.8:45929
--local_ip_for_outbound_sockets=127.20.99.8
--tserver_master_addrs=127.20.99.62:33535
--webserver_port=35637
--webserver_interface=127.20.99.8
--builtin_ntp_servers=127.20.99.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:50.601702 4248 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
I20240503 14:15:50.604923 4514 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:50.605235 4514 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:50.606149 4514 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:50.607604 2151 ts_manager.cc:194] Re-registered known tserver with Master: d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:15:50.608338 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.7:46855
W20240503 14:15:50.969211 4517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:50.969518 4517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:50.969653 4517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:50.977821 4517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:50.977964 4517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:50.978121 4517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.99.8
I20240503 14:15:50.988279 4517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.99.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.99.8:45929
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.99.8
--webserver_port=35637
--tserver_master_addrs=127.20.99.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.99.8
--log_dir=/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:50.989667 4517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:50.991413 4517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:51.003561 4522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:51.004279 4523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:51.005057 4525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:51.005980 4517 server_base.cc:1034] running on GCE node
I20240503 14:15:51.006548 4517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:51.007454 4517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:51.008651 4517 hybrid_clock.cc:648] HybridClock initialized: now 1714745751008603 us; error 45 us; skew 500 ppm
I20240503 14:15:51.008955 4517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:51.011109 4517 webserver.cc:460] Webserver started at http://127.20.99.8:35637/ using document root <none> and password file <none>
I20240503 14:15:51.011838 4517 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:51.011963 4517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:51.017223 4517 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 14:15:51.020051 4531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.021687 4517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:15:51.021888 4517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
format_stamp: "Formatted at 2024-05-03 14:15:32 on dist-test-slave-p0gh"
I20240503 14:15:51.022827 4517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:51.039086 4517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:51.040007 4517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:51.040335 4517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:51.041467 4517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:51.043002 4517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:51.043080 4517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:51.043257 4517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:51.043318 4517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 14:15:51.075598 4517 rpc_server.cc:307] RPC server started. Bound to: 127.20.99.8:45929
I20240503 14:15:51.075635 4643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.99.8:45929 every 8 connection(s)
I20240503 14:15:51.077059 4517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:15:51.085824 20876 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu as pid 4517
I20240503 14:15:51.090008 4644 heartbeater.cc:347] Connected to a master server at 127.20.99.62:33535
I20240503 14:15:51.090309 4644 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:51.091070 4644 heartbeater.cc:511] Master 127.20.99.62:33535 requested a full tablet report, sending...
I20240503 14:15:51.092465 2151 ts_manager.cc:194] Re-registered known tserver with Master: c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:15:51.093348 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.99.8:57767
I20240503 14:15:51.110157 4380 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
I20240503 14:15:51.611243 4514 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
I20240503 14:15:52.095695 4644 heartbeater.cc:503] Master 127.20.99.62:33535 was elected leader, sending a full tablet report...
I20240503 14:15:52.308809 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.321442 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:52.367197 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.395658 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:52.422159 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.437870 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:52.463560 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.476850 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:52.488850 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.506932 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:52.510953 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.536190 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:52.552465 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:52.579979 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.582226 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:52.600557 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.847482 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.892872 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:52.931072 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:52.974975 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:53.011545 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.040330 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.052927 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:53.070458 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.076786 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:53.085290 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:53.100234 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:53.144076 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.173393 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.178047 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:53.218986 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.226588 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:53.569749 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:53.613513 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:53.614602 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:53.630931 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.632921 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.657042 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:53.678530 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:53.734802 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.045584 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.081696 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:54.096380 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.106249 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.115792 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.142910 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:54.158041 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.188608 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.249075 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.308506 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.328299 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:54.411895 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.424089 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.460194 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:54.465263 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.516002 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.697057 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.728544 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.751947 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.765797 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.776248 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.800446 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.808418 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:15:54.811300 2235 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 69481.1us
I20240503 14:15:54.836334 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.848671 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.876991 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:54.896010 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:54.897313 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.898615 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:54.932422 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:54.971843 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:54.989512 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.171788 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.186443 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.222901 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:55.223242 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:55.232542 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:55.273391 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.316921 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:55.366654 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.534438 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.564761 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:55.569161 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:55.592118 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:55.611644 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.619673 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:55.628058 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:55.667256 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.075557 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.102075 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.120849 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.136485 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.140817 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.142982 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:56.200239 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.253512 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.289745 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.294085 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.324782 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:56.330678 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.357985 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.394510 2301 consensus_queue.cc:237] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:56.404516 2561 raft_consensus.cc:1261] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.405297 5289 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.408707 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.411774 2431 raft_consensus.cc:1261] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.413355 5289 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.433777 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:56.464331 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.473646 5289 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:56.478204 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.479086 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:56.479660 2560 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:56.489459 2152 catalog_manager.cc:5484] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:56.490202 2432 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
W20240503 14:15:56.531347 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 944d2a983d6247498b02a6e7b7daf746. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:56.563105 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:56.580826 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.597954 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:56.632844 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.652228 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.652554 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.764533 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.799743 5396 ts_tablet_manager.cc:902] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:56.807214 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.813019 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:56.838666 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:56.851577 5396 tablet_copy_client.cc:308] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:15:56.857282 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.857355 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.872506 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:56.873735 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:56.876214 2300 consensus_queue.cc:237] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:56.918762 2560 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.919512 5289 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:56.924389 2237 consensus_peers.cc:485] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d584b7026b64ba1b720ed21d7019740. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:56.928516 2321 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet 944d2a983d6247498b02a6e7b7daf746 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:56357)
I20240503 14:15:56.928822 2321 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet 944d2a983d6247498b02a6e7b7daf746 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:56357: session id = 7146de39e348442da4a363e95fbe0789-944d2a983d6247498b02a6e7b7daf746
I20240503 14:15:56.931844 2321 tablet_copy_source_session.cc:205] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:56.937608 5396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d2a983d6247498b02a6e7b7daf746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.940887 2690 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:56.941578 5289 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:56.956310 5396 tablet_copy_client.cc:783] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:56.956759 5396 tablet_copy_client.cc:647] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:56.959262 5396 tablet_copy_client.cc:523] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:56.965296 5396 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:15:56.980348 5396 log.cc:824] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:56.985466 5289 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:56.988930 2691 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:56.992525 2560 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:56.997643 2155 catalog_manager.cc:5484] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:57.067498 5396 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:57.068271 5396 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:15:57.082614 5396 ts_tablet_manager.cc:1367] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.118s user 0.076s sys 0.002s
I20240503 14:15:57.119505 5396 raft_consensus.cc:348] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:57.124249 5396 raft_consensus.cc:729] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:15:57.128058 5396 consensus_queue.cc:260] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:57.139786 5396 ts_tablet_manager.cc:1398] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.057s user 0.009s sys 0.003s
I20240503 14:15:57.145430 2321 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-944d2a983d6247498b02a6e7b7daf746 received from {username='slave'} at 127.20.99.5:56357
I20240503 14:15:57.145612 2321 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session 7146de39e348442da4a363e95fbe0789-944d2a983d6247498b02a6e7b7daf746 on tablet 944d2a983d6247498b02a6e7b7daf746 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:15:57.307482 2431 consensus_queue.cc:237] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:57.320776 2300 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.323890 5377 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.327942 2690 consensus_queue.cc:237] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:57.328872 2560 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.329586 5377 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:57.368005 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c94404d8d4f45d59958ae6feb36fc24. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.370325 2300 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.371022 5211 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.376008 5482 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:57.384584 2560 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.385299 5211 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.400099 2561 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:57.401405 4203 raft_consensus.cc:1203] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:57.407599 2159 catalog_manager.cc:5484] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:57.409406 5211 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.415287 2301 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:57.419924 2300 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.423157 2560 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.425344 2155 catalog_manager.cc:5484] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:15:57.467962 2625 consensus_peers.cc:485] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd294a04366145378e3f5e3d306152e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.481559 2300 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:57.495045 2431 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.496091 5289 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.498708 5518 ts_tablet_manager.cc:902] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:57.520805 2690 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.523568 5289 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:57.562561 2234 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf4675882e64f35bc3dc9ee06ae366d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.564008 2690 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.571157 5518 tablet_copy_client.cc:308] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:15:57.593915 2153 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.563040 5530 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.644626 2321 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet 1d584b7026b64ba1b720ed21d7019740 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46371)
I20240503 14:15:57.644857 2321 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet 1d584b7026b64ba1b720ed21d7019740 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46371: session id = c2cf5d2244244a2db5ebc7dc46fd430d-1d584b7026b64ba1b720ed21d7019740
I20240503 14:15:57.651261 2321 tablet_copy_source_session.cc:205] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:57.656384 5518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d584b7026b64ba1b720ed21d7019740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:57.670625 5518 tablet_copy_client.cc:783] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:57.671016 5518 tablet_copy_client.cc:647] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:57.702857 5518 tablet_copy_client.cc:523] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:57.712603 5518 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:15:57.727809 2431 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.766532 2690 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:57.779026 2560 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.782088 2431 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.782968 5573 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.783432 5213 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.789803 5518 log.cc:824] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:57.810465 5573 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:15:57.811465 2628 consensus_peers.cc:485] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1d30b76c4d43d0a54a49892d09a6f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.815570 2560 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:57.817997 2300 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:57.820636 2431 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:57.821115 2432 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.821957 5550 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.825156 2690 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:15:57.825381 2234 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75bafc7675ca45639b696db6e4b7ce36. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:57.826995 2153 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:57.829694 5550 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.881546 5550 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.885383 2691 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.888435 2432 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.891577 2155 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:57.925613 2691 consensus_queue.cc:237] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:57.926627 5601 ts_tablet_manager.cc:902] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:57.928675 5604 ts_tablet_manager.cc:902] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:57.949360 2432 raft_consensus.cc:1261] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.950160 5573 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.964737 2560 raft_consensus.cc:1261] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:57.965513 5573 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:57.970429 5615 ts_tablet_manager.cc:902] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:58.011111 5601 tablet_copy_client.cc:308] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:15:58.021113 5573 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.023908 2560 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.028088 2432 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.031129 2159 catalog_manager.cc:5484] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.048532 5615 tablet_copy_client.cc:308] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
W20240503 14:15:58.062471 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9974d80e8d31417788e10140de97e722. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:58.081822 5604 tablet_copy_client.cc:308] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:15:58.139392 2321 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:43323)
I20240503 14:15:58.139592 2321 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:43323: session id = d4cbc83202c8473d8c897582d30e5989-8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:15:58.142603 2321 tablet_copy_source_session.cc:205] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.149066 5604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.156262 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet dd294a04366145378e3f5e3d306152e3 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:57913)
I20240503 14:15:58.156525 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet dd294a04366145378e3f5e3d306152e3 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:57913: session id = d4cbc83202c8473d8c897582d30e5989-dd294a04366145378e3f5e3d306152e3
I20240503 14:15:58.160677 2712 tablet_copy_source_session.cc:205] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.162671 5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd294a04366145378e3f5e3d306152e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.164827 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 6c94404d8d4f45d59958ae6feb36fc24 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46595)
I20240503 14:15:58.165158 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 6c94404d8d4f45d59958ae6feb36fc24 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46595: session id = c2cf5d2244244a2db5ebc7dc46fd430d-6c94404d8d4f45d59958ae6feb36fc24
I20240503 14:15:58.171691 2452 tablet_copy_source_session.cc:205] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.174223 5601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c94404d8d4f45d59958ae6feb36fc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.182755 5601 tablet_copy_client.cc:783] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.183224 5601 tablet_copy_client.cc:647] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.185827 5601 tablet_copy_client.cc:523] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.195549 5662 ts_tablet_manager.cc:902] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:58.199093 5601 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:15:58.213078 5604 tablet_copy_client.cc:783] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.220108 5604 tablet_copy_client.cc:647] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.224512 5604 tablet_copy_client.cc:523] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.232641 5615 tablet_copy_client.cc:783] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.233062 5615 tablet_copy_client.cc:647] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.233525 5518 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.234136 5518 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:15:58.235067 5662 tablet_copy_client.cc:308] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:15:58.237675 5518 ts_tablet_manager.cc:1367] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.525s user 0.075s sys 0.000s
I20240503 14:15:58.241163 5604 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:15:58.241451 5615 tablet_copy_client.cc:523] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.245173 5615 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:15:58.258226 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet eb1d30b76c4d43d0a54a49892d09a6f8 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:38575)
I20240503 14:15:58.258464 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet eb1d30b76c4d43d0a54a49892d09a6f8 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:38575: session id = 285488446ac445c5a5dfe654bfaf9835-eb1d30b76c4d43d0a54a49892d09a6f8
I20240503 14:15:58.261349 5518 raft_consensus.cc:348] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:58.261862 5518 raft_consensus.cc:729] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.262162 2712 tablet_copy_source_session.cc:205] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.267900 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1d30b76c4d43d0a54a49892d09a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.269152 5518 consensus_queue.cc:260] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:58.281795 5518 ts_tablet_manager.cc:1398] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.044s user 0.011s sys 0.000s
I20240503 14:15:58.287680 2321 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-1d584b7026b64ba1b720ed21d7019740 received from {username='slave'} at 127.20.99.8:46371
I20240503 14:15:58.287900 2321 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-1d584b7026b64ba1b720ed21d7019740 on tablet 1d584b7026b64ba1b720ed21d7019740 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:15:58.295462 5662 tablet_copy_client.cc:783] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.295960 5662 tablet_copy_client.cc:647] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.299461 5662 tablet_copy_client.cc:523] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.308494 5662 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:15:58.290330 5604 log.cc:824] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:58.354175 2691 consensus_queue.cc:237] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:58.356842 5662 log.cc:824] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:58.381335 2560 raft_consensus.cc:1261] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:58.382064 5573 consensus_queue.cc:1035] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:58.397989 2300 raft_consensus.cc:1261] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:58.398620 5573 consensus_queue.cc:1035] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:58.422479 5213 raft_consensus.cc:2932] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.423317 2561 raft_consensus.cc:2932] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.429203 2300 raft_consensus.cc:2932] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.431164 2153 catalog_manager.cc:5484] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.452649 5706 ts_tablet_manager.cc:902] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:58.503738 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:15:58.547138 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff59573e2c8f472585b96ca0a0fd75b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:58.560314 5518 ts_tablet_manager.cc:902] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:58.567497 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.589071 5518 tablet_copy_client.cc:308] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:15:58.594532 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:58.600666 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 9974d80e8d31417788e10140de97e722 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:41375)
I20240503 14:15:58.600844 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 9974d80e8d31417788e10140de97e722 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:41375: session id = c2cf5d2244244a2db5ebc7dc46fd430d-9974d80e8d31417788e10140de97e722
I20240503 14:15:58.600981 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:58.604375 2712 tablet_copy_source_session.cc:205] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.609994 5518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9974d80e8d31417788e10140de97e722. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.617180 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.626920 5518 tablet_copy_client.cc:783] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.630390 4599 raft_consensus.cc:1203] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:58.643692 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:58.667658 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.668314 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:58.672701 5518 tablet_copy_client.cc:647] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.677912 5601 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.678414 5601 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:15:58.678706 5601 ts_tablet_manager.cc:1367] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.480s user 0.070s sys 0.011s
I20240503 14:15:58.679713 5601 raft_consensus.cc:348] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:58.679971 5601 raft_consensus.cc:729] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.689106 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.694748 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.701833 5518 tablet_copy_client.cc:523] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.713819 5518 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:15:58.715495 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:15:58.727870 2432 consensus_queue.cc:237] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:58.737735 5601 consensus_queue.cc:260] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:58.740585 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:15:58.741446 2431 consensus_queue.cc:237] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:58.743162 5601 ts_tablet_manager.cc:1398] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.064s user 0.004s sys 0.000s
I20240503 14:15:58.744520 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.746624 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-6c94404d8d4f45d59958ae6feb36fc24 received from {username='slave'} at 127.20.99.8:46595
I20240503 14:15:58.746775 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-6c94404d8d4f45d59958ae6feb36fc24 on tablet 6c94404d8d4f45d59958ae6feb36fc24 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:15:58.751276 5604 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.759958 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:58.762993 5604 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:15:58.763891 2560 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:58.765852 5481 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:58.767854 5604 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.527s user 0.072s sys 0.006s
I20240503 14:15:58.768648 4599 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:58.769449 5481 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:58.779222 5706 tablet_copy_client.cc:308] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:15:58.780717 2321 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet 75bafc7675ca45639b696db6e4b7ce36 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:43323)
I20240503 14:15:58.780894 2321 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet 75bafc7675ca45639b696db6e4b7ce36 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:43323: session id = d4cbc83202c8473d8c897582d30e5989-75bafc7675ca45639b696db6e4b7ce36
I20240503 14:15:58.785923 2321 tablet_copy_source_session.cc:205] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:58.787357 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75bafc7675ca45639b696db6e4b7ce36. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:15:58.788829 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c94404d8d4f45d59958ae6feb36fc24. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:58.803105 5615 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.806005 5481 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:58.809635 2561 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:58.814078 5604 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:58.814595 5604 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.817716 2159 catalog_manager.cc:5484] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:58.820185 5615 tablet_bootstrap.cc:492] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:15:58.821781 5615 ts_tablet_manager.cc:1367] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.575s user 0.063s sys 0.000s
I20240503 14:15:58.822974 5604 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:58.829093 5615 raft_consensus.cc:348] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:58.831538 5615 raft_consensus.cc:729] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.832996 5774 ts_tablet_manager.cc:902] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:15:58.832818 2300 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:58.834597 5706 tablet_copy_client.cc:783] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:58.838658 5706 tablet_copy_client.cc:647] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:58.839411 5615 consensus_queue.cc:260] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:58.842672 5706 tablet_copy_client.cc:523] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:58.846495 5706 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:15:58.850073 2430 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:58.850656 5550 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:58.852560 2237 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe0eebf47c494dad8edf06197cdf15bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:58.865032 5604 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.095s user 0.010s sys 0.001s
I20240503 14:15:58.865892 2321 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-8bf4675882e64f35bc3dc9ee06ae366d received from {username='slave'} at 127.20.99.7:43323
I20240503 14:15:58.866021 2321 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-8bf4675882e64f35bc3dc9ee06ae366d on tablet 8bf4675882e64f35bc3dc9ee06ae366d with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:15:58.874162 2691 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:58.875460 5780 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:58.886123 5662 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:58.888257 5662 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:15:58.892891 5550 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.899367 2300 raft_consensus.cc:1261] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:58.902843 5377 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:58.906912 2691 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:15:58.917950 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 470b425f5cef494ab3334fe2b85e1c4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:58.919637 2560 raft_consensus.cc:1261] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:58.920830 2431 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.924093 2159 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:15:58.927147 5377 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:58.938197 5662 ts_tablet_manager.cc:1367] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.630s user 0.079s sys 0.001s
I20240503 14:15:58.955163 5377 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.958655 5774 tablet_copy_client.cc:308] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:15:58.973299 5662 raft_consensus.cc:348] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:58.973829 5662 raft_consensus.cc:729] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.974673 5662 consensus_queue.cc:260] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:58.981745 2300 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:58.982751 5798 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:58.982720 2300 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.992298 2159 catalog_manager.cc:5484] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:58.992425 4599 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:58.997856 5615 ts_tablet_manager.cc:1398] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.176s user 0.029s sys 0.000s
I20240503 14:15:58.998724 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-dd294a04366145378e3f5e3d306152e3 received from {username='slave'} at 127.20.99.7:57913
I20240503 14:15:58.998893 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-dd294a04366145378e3f5e3d306152e3 on tablet dd294a04366145378e3f5e3d306152e3 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:15:59.007339 2561 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:59.009325 5518 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.009749 5518 tablet_bootstrap.cc:492] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:15:59.010013 5518 ts_tablet_manager.cc:1367] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.296s user 0.053s sys 0.004s
I20240503 14:15:59.010779 5518 raft_consensus.cc:348] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:59.011044 5518 raft_consensus.cc:729] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.011276 5518 consensus_queue.cc:260] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:59.017424 5518 ts_tablet_manager.cc:1398] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:15:59.020135 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-9974d80e8d31417788e10140de97e722 received from {username='slave'} at 127.20.99.8:41375
I20240503 14:15:59.020285 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-9974d80e8d31417788e10140de97e722 on tablet 9974d80e8d31417788e10140de97e722 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:15:59.025689 4599 raft_consensus.cc:1203] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:59.042955 2301 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:59.046617 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:59.049328 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:15:59.060070 5706 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.060714 5706 tablet_bootstrap.cc:492] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:15:59.061044 5706 ts_tablet_manager.cc:1367] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.215s user 0.070s sys 0.044s
I20240503 14:15:59.061899 5706 raft_consensus.cc:348] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:59.062163 5706 raft_consensus.cc:729] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.062480 5706 consensus_queue.cc:260] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:59.063302 5706 ts_tablet_manager.cc:1398] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:59.064030 2321 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-75bafc7675ca45639b696db6e4b7ce36 received from {username='slave'} at 127.20.99.7:43323
I20240503 14:15:59.064198 2321 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-75bafc7675ca45639b696db6e4b7ce36 on tablet 75bafc7675ca45639b696db6e4b7ce36 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:15:59.093247 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet ff59573e2c8f472585b96ca0a0fd75b1 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:59251)
I20240503 14:15:59.093466 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet ff59573e2c8f472585b96ca0a0fd75b1 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:59251: session id = 7146de39e348442da4a363e95fbe0789-ff59573e2c8f472585b96ca0a0fd75b1
I20240503 14:15:59.097767 2712 tablet_copy_source_session.cc:205] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:59.099218 5774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff59573e2c8f472585b96ca0a0fd75b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.110150 5774 tablet_copy_client.cc:783] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:59.110538 5774 tablet_copy_client.cc:647] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:59.112159 5662 ts_tablet_manager.cc:1398] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.174s user 0.005s sys 0.006s
I20240503 14:15:59.113797 5774 tablet_copy_client.cc:523] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:59.115346 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-eb1d30b76c4d43d0a54a49892d09a6f8 received from {username='slave'} at 127.20.99.6:38575
I20240503 14:15:59.115594 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-eb1d30b76c4d43d0a54a49892d09a6f8 on tablet eb1d30b76c4d43d0a54a49892d09a6f8 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:15:59.118839 5774 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:15:59.135725 4335 raft_consensus.cc:1203] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:59.217384 4469 raft_consensus.cc:1203] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:59.224071 5774 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.225111 5774 tablet_bootstrap.cc:492] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:15:59.225539 5774 ts_tablet_manager.cc:1367] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.107s user 0.052s sys 0.008s
I20240503 14:15:59.227047 5774 raft_consensus.cc:348] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:59.227538 5774 raft_consensus.cc:729] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.227891 5774 consensus_queue.cc:260] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:59.281920 5774 ts_tablet_manager.cc:1398] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.056s user 0.028s sys 0.011s
I20240503 14:15:59.286587 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-ff59573e2c8f472585b96ca0a0fd75b1 received from {username='slave'} at 127.20.99.5:59251
I20240503 14:15:59.286726 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 7146de39e348442da4a363e95fbe0789-ff59573e2c8f472585b96ca0a0fd75b1 on tablet ff59573e2c8f472585b96ca0a0fd75b1 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:15:59.311385 5774 ts_tablet_manager.cc:902] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:59.314955 5662 ts_tablet_manager.cc:902] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:15:59.323880 4469 raft_consensus.cc:1203] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:59.331072 2561 consensus_queue.cc:237] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:15:59.346932 2560 consensus_queue.cc:237] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:59.358331 4203 raft_consensus.cc:1203] T ff59573e2c8f472585b96ca0a0fd75b1 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:59.375579 2301 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.376596 5703 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.386659 5774 tablet_copy_client.cc:308] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:15:59.399230 2299 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.399967 5703 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.400950 2432 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.401505 5703 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.410775 2431 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.411502 5703 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.426363 5662 tablet_copy_client.cc:308] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:15:59.432332 5518 ts_tablet_manager.cc:902] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:15:59.433537 4469 raft_consensus.cc:1203] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:59.438813 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 470b425f5cef494ab3334fe2b85e1c4e from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:56485)
I20240503 14:15:59.439018 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 470b425f5cef494ab3334fe2b85e1c4e from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:56485: session id = 7146de39e348442da4a363e95fbe0789-470b425f5cef494ab3334fe2b85e1c4e
I20240503 14:15:59.442060 2452 tablet_copy_source_session.cc:205] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:59.444686 5774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b425f5cef494ab3334fe2b85e1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.451572 5774 tablet_copy_client.cc:783] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:59.451814 5774 tablet_copy_client.cc:647] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:59.453653 5774 tablet_copy_client.cc:523] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:59.457248 5774 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
W20240503 14:15:59.484496 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aba6537cdb87400cb8b108b1fbfa6c0f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:59.502175 5869 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:59.504280 5518 tablet_copy_client.cc:308] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:15:59.505513 2321 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet fe0eebf47c494dad8edf06197cdf15bc from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46371)
I20240503 14:15:59.505725 2321 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet fe0eebf47c494dad8edf06197cdf15bc from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46371: session id = c2cf5d2244244a2db5ebc7dc46fd430d-fe0eebf47c494dad8edf06197cdf15bc
I20240503 14:15:59.509851 2321 tablet_copy_source_session.cc:205] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:59.512498 5518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0eebf47c494dad8edf06197cdf15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.516237 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 6c94404d8d4f45d59958ae6feb36fc24 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:52553)
I20240503 14:15:59.516440 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 6c94404d8d4f45d59958ae6feb36fc24 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:52553: session id = 285488446ac445c5a5dfe654bfaf9835-6c94404d8d4f45d59958ae6feb36fc24
I20240503 14:15:59.520864 2452 tablet_copy_source_session.cc:205] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:59.522060 5518 tablet_copy_client.cc:783] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:59.522336 5518 tablet_copy_client.cc:647] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:59.526551 5518 tablet_copy_client.cc:523] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:59.533239 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c94404d8d4f45d59958ae6feb36fc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.534933 5518 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:15:59.542850 5662 tablet_copy_client.cc:783] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:59.543130 5662 tablet_copy_client.cc:647] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:59.547299 5662 tablet_copy_client.cc:523] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:59.551052 5662 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
W20240503 14:15:59.565241 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 78378.6us
I20240503 14:15:59.567905 5868 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
W20240503 14:15:59.583546 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57c4a133b5c646f78b07f37dc85904f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:59.596397 2159 catalog_manager.cc:5484] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:59.598055 2431 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:59.602197 2301 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:59.603057 2299 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:15:59.612077 2153 catalog_manager.cc:5484] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:59.637954 2431 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:15:59.720562 5774 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.721105 5774 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:15:59.721427 5774 ts_tablet_manager.cc:1367] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.264s user 0.057s sys 0.012s
I20240503 14:15:59.722525 5774 raft_consensus.cc:348] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:59.738014 5774 raft_consensus.cc:729] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.738415 5774 consensus_queue.cc:260] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:15:59.747752 5774 ts_tablet_manager.cc:1398] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.026s user 0.003s sys 0.001s
I20240503 14:15:59.765621 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-470b425f5cef494ab3334fe2b85e1c4e received from {username='slave'} at 127.20.99.5:56485
I20240503 14:15:59.765825 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 7146de39e348442da4a363e95fbe0789-470b425f5cef494ab3334fe2b85e1c4e on tablet 470b425f5cef494ab3334fe2b85e1c4e with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:15:59.773041 2690 consensus_queue.cc:237] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:59.784039 5518 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.784571 5518 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:15:59.784860 5518 ts_tablet_manager.cc:1367] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.250s user 0.055s sys 0.006s
I20240503 14:15:59.785660 5518 raft_consensus.cc:348] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:15:59.794598 5518 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.795203 5518 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:15:59.800894 2628 consensus_peers.cc:485] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b112f8fb89c7404b87a4c6e3d72fdcfd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:59.801450 2299 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.801690 2431 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:59.802918 5573 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.803721 5573 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:59.820636 5518 ts_tablet_manager.cc:1398] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.036s user 0.003s sys 0.003s
I20240503 14:15:59.822763 5213 raft_consensus.cc:2932] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:59.823267 2321 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-fe0eebf47c494dad8edf06197cdf15bc received from {username='slave'} at 127.20.99.8:46371
I20240503 14:15:59.823446 2321 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-fe0eebf47c494dad8edf06197cdf15bc on tablet fe0eebf47c494dad8edf06197cdf15bc with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:15:59.837108 2299 raft_consensus.cc:2932] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:59.840371 2301 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:59.842972 5662 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:59.843447 5662 tablet_bootstrap.cc:492] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:15:59.843729 5662 ts_tablet_manager.cc:1367] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.293s user 0.062s sys 0.010s
I20240503 14:15:59.844897 5662 raft_consensus.cc:348] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:59.845229 5662 raft_consensus.cc:729] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.845624 5662 consensus_queue.cc:260] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:15:59.848722 5774 ts_tablet_manager.cc:902] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:15:59.860994 5774 tablet_copy_client.cc:308] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:15:59.871906 5662 ts_tablet_manager.cc:1398] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.028s user 0.017s sys 0.010s
I20240503 14:15:59.873049 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 57c4a133b5c646f78b07f37dc85904f6 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:39003)
I20240503 14:15:59.873313 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 57c4a133b5c646f78b07f37dc85904f6 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:39003: session id = 7146de39e348442da4a363e95fbe0789-57c4a133b5c646f78b07f37dc85904f6
I20240503 14:15:59.876611 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-6c94404d8d4f45d59958ae6feb36fc24 received from {username='slave'} at 127.20.99.6:52553
I20240503 14:15:59.876768 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-6c94404d8d4f45d59958ae6feb36fc24 on tablet 6c94404d8d4f45d59958ae6feb36fc24 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:15:59.879886 2582 tablet_copy_source_session.cc:205] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:59.884476 2152 catalog_manager.cc:5484] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:15:59.884935 5774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c4a133b5c646f78b07f37dc85904f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.891219 5774 tablet_copy_client.cc:783] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:59.891480 5774 tablet_copy_client.cc:647] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:59.910326 5774 tablet_copy_client.cc:523] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:59.915518 5774 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:15:59.923787 2431 raft_consensus.cc:2932] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:15:59.940248 2234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:44387, user_credentials={real_user=slave}} blocked reactor thread for 96090.7us
I20240503 14:15:59.941578 2690 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:15:59.942404 2626 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 98013.4us
I20240503 14:15:59.948606 5789 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.965687 4599 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.969934 5789 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.979684 2432 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:59.980443 5947 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:59.989434 5969 ts_tablet_manager.cc:902] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:00.001662 2690 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:00.004607 2301 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:16:00.004885 2625 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abdff0b719a342cfbde746751bd2a89d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.005354 5573 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.005614 2561 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.006232 5213 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:16:00.015743 2366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 72254.6us
I20240503 14:16:00.015537 5961 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.020330 2432 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.022804 5969 tablet_copy_client.cc:308] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:00.030413 4599 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.030856 5573 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:00.033926 2301 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:00.038090 2561 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:00.038414 2152 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:16:00.053586 2237 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe0eebf47c494dad8edf06197cdf15bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.054543 2690 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.056034 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet aba6537cdb87400cb8b108b1fbfa6c0f from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:41209)
I20240503 14:16:00.056246 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet aba6537cdb87400cb8b108b1fbfa6c0f from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:41209: session id = d4cbc83202c8473d8c897582d30e5989-aba6537cdb87400cb8b108b1fbfa6c0f
I20240503 14:16:00.059937 2582 tablet_copy_source_session.cc:205] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:00.062577 5969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba6537cdb87400cb8b108b1fbfa6c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.072093 2152 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.092538 5969 tablet_copy_client.cc:783] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:00.093068 5969 tablet_copy_client.cc:647] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:00.095659 5969 tablet_copy_client.cc:523] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:00.101619 5969 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:00.199915 5969 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:00.200512 5969 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:00.200939 5969 ts_tablet_manager.cc:1367] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.099s user 0.059s sys 0.008s
I20240503 14:16:00.201840 5969 raft_consensus.cc:348] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:00.202242 5969 raft_consensus.cc:729] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.202631 5969 consensus_queue.cc:260] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:00.211128 5969 ts_tablet_manager.cc:1398] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240503 14:16:00.212177 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-aba6537cdb87400cb8b108b1fbfa6c0f received from {username='slave'} at 127.20.99.7:41209
I20240503 14:16:00.212320 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-aba6537cdb87400cb8b108b1fbfa6c0f on tablet aba6537cdb87400cb8b108b1fbfa6c0f with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:00.216811 2299 consensus_queue.cc:237] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:00.220050 4203 raft_consensus.cc:1203] T 470b425f5cef494ab3334fe2b85e1c4e P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:00.230692 2690 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:16:00.232393 2237 consensus_peers.cc:485] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ae190d0f4d843c990e6fdadcac677ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.232781 5944 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.233168 2432 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.235026 5961 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.249481 5961 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.255820 5774 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:00.257968 2690 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.258622 2432 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.260694 2159 catalog_manager.cc:5484] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:00.266489 5774 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:00.269721 5774 ts_tablet_manager.cc:1367] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.354s user 0.053s sys 0.019s
I20240503 14:16:00.271093 5774 raft_consensus.cc:348] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:00.271452 5774 raft_consensus.cc:729] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.271759 5774 consensus_queue.cc:260] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:00.275378 5774 ts_tablet_manager.cc:1398] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:16:00.278769 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-57c4a133b5c646f78b07f37dc85904f6 received from {username='slave'} at 127.20.99.5:39003
I20240503 14:16:00.278937 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 7146de39e348442da4a363e95fbe0789-57c4a133b5c646f78b07f37dc85904f6 on tablet 57c4a133b5c646f78b07f37dc85904f6 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:00.336555 4334 raft_consensus.cc:1203] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:00.351593 5662 ts_tablet_manager.cc:902] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:00.380682 4203 raft_consensus.cc:1203] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:00.445187 6065 ts_tablet_manager.cc:902] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:16:00.461891 5662 tablet_copy_client.cc:308] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:00.462867 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet b112f8fb89c7404b87a4c6e3d72fdcfd from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:38575)
I20240503 14:16:00.463069 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet b112f8fb89c7404b87a4c6e3d72fdcfd from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:38575: session id = 285488446ac445c5a5dfe654bfaf9835-b112f8fb89c7404b87a4c6e3d72fdcfd
I20240503 14:16:00.470188 5969 ts_tablet_manager.cc:902] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:00.472041 2432 consensus_queue.cc:237] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:00.478116 2690 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:00.485515 2712 tablet_copy_source_session.cc:205] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:00.487270 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b112f8fb89c7404b87a4c6e3d72fdcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.494788 5662 tablet_copy_client.cc:783] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:00.495018 5662 tablet_copy_client.cc:647] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:00.499845 5662 tablet_copy_client.cc:523] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:00.503602 5662 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
W20240503 14:16:00.532502 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b63f87fb384bb2a48044bcad87c282. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.533975 2431 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.534485 2561 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.535449 5573 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.536123 5573 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:16:00.547473 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc88c5dc94844f1c8400c31ad1165a32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:00.550740 2560 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.551499 5968 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.558362 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:00.568141 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:00.584626 4469 raft_consensus.cc:1203] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:00.594230 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:00.598125 2691 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:00.599043 5968 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:00.650692 5946 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:00.680261 2432 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:00.684579 2159 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:00.721115 2560 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:00.732987 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:00.798156 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:00.816947 6095 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:00.822160 6096 ts_tablet_manager.cc:902] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:16:00.823098 2691 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:00.831557 2159 catalog_manager.cc:5484] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:16:00.843089 2627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 85409.2us
I20240503 14:16:00.852123 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:16:00.868108 2366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 111185us
I20240503 14:16:00.870106 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:00.870774 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:00.878765 5969 tablet_copy_client.cc:308] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:00.880369 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet abdff0b719a342cfbde746751bd2a89d from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:57913)
I20240503 14:16:00.880565 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet abdff0b719a342cfbde746751bd2a89d from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:57913: session id = d4cbc83202c8473d8c897582d30e5989-abdff0b719a342cfbde746751bd2a89d
I20240503 14:16:00.883724 2712 tablet_copy_source_session.cc:205] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:00.890303 5969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdff0b719a342cfbde746751bd2a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.929793 5969 tablet_copy_client.cc:783] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:00.930258 5969 tablet_copy_client.cc:647] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:00.933023 5969 tablet_copy_client.cc:523] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:00.939165 5969 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:00.939996 6096 tablet_copy_client.cc:308] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:16:00.954452 2560 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:01.008414 6065 tablet_copy_client.cc:308] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:16:01.029592 6126 ts_tablet_manager.cc:902] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:01.038579 6127 ts_tablet_manager.cc:902] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:01.085244 2321 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet 9ae190d0f4d843c990e6fdadcac677ed from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:48475)
I20240503 14:16:01.085462 2321 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet 9ae190d0f4d843c990e6fdadcac677ed from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:48475: session id = 285488446ac445c5a5dfe654bfaf9835-9ae190d0f4d843c990e6fdadcac677ed
I20240503 14:16:01.088557 2321 tablet_copy_source_session.cc:205] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.089129 2320 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet fe0eebf47c494dad8edf06197cdf15bc from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:48475)
I20240503 14:16:01.089258 2320 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet fe0eebf47c494dad8edf06197cdf15bc from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:48475: session id = 285488446ac445c5a5dfe654bfaf9835-fe0eebf47c494dad8edf06197cdf15bc
I20240503 14:16:01.095695 6096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae190d0f4d843c990e6fdadcac677ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.101012 5662 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.103029 2320 tablet_copy_source_session.cc:205] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.109244 5662 tablet_bootstrap.cc:492] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:01.109688 5662 ts_tablet_manager.cc:1367] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.606s user 0.112s sys 0.060s
I20240503 14:16:01.110183 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0eebf47c494dad8edf06197cdf15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.110736 5662 raft_consensus.cc:348] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.111078 5662 raft_consensus.cc:729] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:01.111429 5662 consensus_queue.cc:260] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.121551 6096 tablet_copy_client.cc:783] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.122116 6096 tablet_copy_client.cc:647] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.121811 6065 tablet_copy_client.cc:783] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.122749 6065 tablet_copy_client.cc:647] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.125684 6065 tablet_copy_client.cc:523] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.126297 6096 tablet_copy_client.cc:523] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.131886 5662 ts_tablet_manager.cc:1398] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20240503 14:16:01.135949 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-b112f8fb89c7404b87a4c6e3d72fdcfd received from {username='slave'} at 127.20.99.6:38575
I20240503 14:16:01.136075 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-b112f8fb89c7404b87a4c6e3d72fdcfd on tablet b112f8fb89c7404b87a4c6e3d72fdcfd with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:01.139609 6096 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:01.149847 6127 tablet_copy_client.cc:308] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:01.150863 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet dc88c5dc94844f1c8400c31ad1165a32 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46595)
I20240503 14:16:01.151096 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet dc88c5dc94844f1c8400c31ad1165a32 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46595: session id = c2cf5d2244244a2db5ebc7dc46fd430d-dc88c5dc94844f1c8400c31ad1165a32
I20240503 14:16:01.154810 2452 tablet_copy_source_session.cc:205] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.157872 6065 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:01.163224 6127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc88c5dc94844f1c8400c31ad1165a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.172123 6126 tablet_copy_client.cc:308] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:01.173281 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 61b63f87fb384bb2a48044bcad87c282 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:59251)
I20240503 14:16:01.173452 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 61b63f87fb384bb2a48044bcad87c282 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:59251: session id = 7146de39e348442da4a363e95fbe0789-61b63f87fb384bb2a48044bcad87c282
I20240503 14:16:01.176788 2712 tablet_copy_source_session.cc:205] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.190416 6126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b63f87fb384bb2a48044bcad87c282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.228415 6127 tablet_copy_client.cc:783] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.229005 6127 tablet_copy_client.cc:647] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.229674 6126 tablet_copy_client.cc:783] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.230494 6126 tablet_copy_client.cc:647] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.232661 6127 tablet_copy_client.cc:523] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.236809 5969 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.237394 6126 tablet_copy_client.cc:523] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.241513 5969 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:01.241832 5969 ts_tablet_manager.cc:1367] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.303s user 0.051s sys 0.005s
I20240503 14:16:01.246932 5969 raft_consensus.cc:348] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:01.247236 5969 raft_consensus.cc:729] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:01.247543 5969 consensus_queue.cc:260] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:01.263563 5969 ts_tablet_manager.cc:1398] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.022s user 0.000s sys 0.004s
I20240503 14:16:01.264876 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-abdff0b719a342cfbde746751bd2a89d received from {username='slave'} at 127.20.99.7:57913
I20240503 14:16:01.265040 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-abdff0b719a342cfbde746751bd2a89d on tablet abdff0b719a342cfbde746751bd2a89d with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:01.267262 6096 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.267720 6096 tablet_bootstrap.cc:492] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:01.268018 6096 ts_tablet_manager.cc:1367] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.128s user 0.043s sys 0.014s
I20240503 14:16:01.270169 6126 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:01.270306 6127 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:01.272264 6096 raft_consensus.cc:348] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.333292 6096 raft_consensus.cc:729] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:01.336503 6096 consensus_queue.cc:260] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.341531 2691 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:16:01.352733 2628 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abdff0b719a342cfbde746751bd2a89d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:01.352792 2560 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.353626 6098 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.358466 2561 consensus_queue.cc:237] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.360769 4469 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.363504 5573 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.371793 2301 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.373207 6098 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.389328 6096 ts_tablet_manager.cc:1398] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.072s user 0.000s sys 0.003s
I20240503 14:16:01.390789 2320 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-9ae190d0f4d843c990e6fdadcac677ed received from {username='slave'} at 127.20.99.6:48475
I20240503 14:16:01.390975 2320 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-9ae190d0f4d843c990e6fdadcac677ed on tablet 9ae190d0f4d843c990e6fdadcac677ed with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:01.393980 2432 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.394734 5868 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.403050 6098 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:01.409533 2301 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:01.413311 4468 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:01.418685 2560 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:01.424944 2159 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:01.439500 4334 raft_consensus.cc:1203] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:01.440536 2299 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.441555 4468 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:01.442008 5868 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.444337 5868 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:01.475581 6127 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.476577 6127 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:01.478320 6127 ts_tablet_manager.cc:1367] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.208s user 0.074s sys 0.021s
I20240503 14:16:01.479804 6127 raft_consensus.cc:348] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:01.482025 6127 raft_consensus.cc:729] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:01.482662 6127 consensus_queue.cc:260] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:01.498533 6127 ts_tablet_manager.cc:1398] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.019s user 0.002s sys 0.004s
I20240503 14:16:01.502321 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-dc88c5dc94844f1c8400c31ad1165a32 received from {username='slave'} at 127.20.99.8:46595
I20240503 14:16:01.502528 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-dc88c5dc94844f1c8400c31ad1165a32 on tablet dc88c5dc94844f1c8400c31ad1165a32 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:01.507866 6215 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:01.510885 4468 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:01.516857 2159 catalog_manager.cc:5484] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:16:01.545024 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 62356.3us
I20240503 14:16:01.549881 2301 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:16:01.551357 2498 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aba6537cdb87400cb8b108b1fbfa6c0f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:01.559044 2432 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:01.651026 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:01.699196 6065 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.699663 6065 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:01.699929 6065 ts_tablet_manager.cc:1367] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.542s user 0.062s sys 0.005s
I20240503 14:16:01.700634 6065 raft_consensus.cc:348] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.700904 6065 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:01.708424 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:01.712673 6126 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:01.713124 6126 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:01.713438 6126 ts_tablet_manager.cc:1367] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.443s user 0.060s sys 0.000s
I20240503 14:16:01.715481 6126 raft_consensus.cc:348] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:01.715790 6126 raft_consensus.cc:729] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:01.716070 6126 consensus_queue.cc:260] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:01.716677 6065 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:01.717558 6065 ts_tablet_manager.cc:1398] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 14:16:01.718302 2320 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-fe0eebf47c494dad8edf06197cdf15bc received from {username='slave'} at 127.20.99.6:48475
I20240503 14:16:01.718472 2320 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-fe0eebf47c494dad8edf06197cdf15bc on tablet fe0eebf47c494dad8edf06197cdf15bc with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:01.743970 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:01.746757 2691 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:01.755345 6126 ts_tablet_manager.cc:1398] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240503 14:16:01.759299 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-61b63f87fb384bb2a48044bcad87c282 received from {username='slave'} at 127.20.99.5:59251
I20240503 14:16:01.759450 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 7146de39e348442da4a363e95fbe0789-61b63f87fb384bb2a48044bcad87c282 on tablet 61b63f87fb384bb2a48044bcad87c282 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:01.772197 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:01.787390 2561 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:16:01.788055 2625 consensus_peers.cc:485] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f139d77b0f0c47809e6408b2771adb73. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:01.788589 6098 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.789477 2301 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.794836 6098 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.821584 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:01.823640 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:01.831578 6127 ts_tablet_manager.cc:902] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:01.848443 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:01.850709 5573 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:01.855712 2560 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:01.858033 2691 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:01.873277 2152 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:01.882524 6065 ts_tablet_manager.cc:902] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:01.886476 2301 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:01.896337 2432 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:16:01.896947 2627 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78edc267e9f44d82b7dbb663628359fe. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:01.898000 2299 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:01.899210 6098 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.899812 6291 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:01.900007 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:01.908325 4334 raft_consensus.cc:1203] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:01.957095 4599 raft_consensus.cc:1203] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:01.962757 6127 tablet_copy_client.cc:308] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:01.963474 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet abdff0b719a342cfbde746751bd2a89d from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:41375)
I20240503 14:16:01.963706 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet abdff0b719a342cfbde746751bd2a89d from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:41375: session id = c2cf5d2244244a2db5ebc7dc46fd430d-abdff0b719a342cfbde746751bd2a89d
I20240503 14:16:01.966920 2712 tablet_copy_source_session.cc:205] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:01.968915 6127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdff0b719a342cfbde746751bd2a89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:01.978484 4203 raft_consensus.cc:1203] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:01.980198 6127 tablet_copy_client.cc:783] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:01.980469 6127 tablet_copy_client.cc:647] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:01.983455 6127 tablet_copy_client.cc:523] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:01.987376 6127 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:02.005651 5573 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:02.013751 2432 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:02.023214 2153 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:02.022948 2299 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:02.074206 2691 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:02.083480 6065 tablet_copy_client.cc:308] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:02.112447 4334 raft_consensus.cc:1203] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:02.129145 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:02.139988 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet aba6537cdb87400cb8b108b1fbfa6c0f from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:59085)
I20240503 14:16:02.140178 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet aba6537cdb87400cb8b108b1fbfa6c0f from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:59085: session id = 285488446ac445c5a5dfe654bfaf9835-aba6537cdb87400cb8b108b1fbfa6c0f
I20240503 14:16:02.143800 2582 tablet_copy_source_session.cc:205] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:02.146788 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba6537cdb87400cb8b108b1fbfa6c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.153276 6065 tablet_copy_client.cc:783] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:02.153510 6065 tablet_copy_client.cc:647] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:02.159641 6065 tablet_copy_client.cc:523] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:02.165216 6065 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:02.166918 6127 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 14:16:02.167126 2628 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fda4460865854505b5220d7ec3b34c57. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:02.167397 6127 tablet_bootstrap.cc:492] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:02.167654 6127 ts_tablet_manager.cc:1367] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.180s user 0.049s sys 0.013s
I20240503 14:16:02.168314 2561 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.168526 6127 raft_consensus.cc:348] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:02.168783 6127 raft_consensus.cc:729] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.169076 6127 consensus_queue.cc:260] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:02.169210 5573 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.169728 6127 ts_tablet_manager.cc:1398] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:16:02.170457 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-abdff0b719a342cfbde746751bd2a89d received from {username='slave'} at 127.20.99.8:41375
I20240503 14:16:02.170572 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-abdff0b719a342cfbde746751bd2a89d on tablet abdff0b719a342cfbde746751bd2a89d with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:02.182890 2299 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.183965 6098 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.216701 5573 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.220868 2299 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.223495 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:02.230082 2159 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.253263 2561 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.309342 4599 raft_consensus.cc:1203] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:02.312067 6338 ts_tablet_manager.cc:902] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:02.338282 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:02.347290 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:02.357100 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:02.357756 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:02.362264 6338 tablet_copy_client.cc:308] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:02.363368 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet f139d77b0f0c47809e6408b2771adb73 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:57913)
I20240503 14:16:02.363550 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet f139d77b0f0c47809e6408b2771adb73 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:57913: session id = d4cbc83202c8473d8c897582d30e5989-f139d77b0f0c47809e6408b2771adb73
I20240503 14:16:02.368471 2712 tablet_copy_source_session.cc:205] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:02.374992 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:02.393855 6338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f139d77b0f0c47809e6408b2771adb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.410609 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:02.443233 6338 tablet_copy_client.cc:783] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:02.443554 6338 tablet_copy_client.cc:647] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:02.444221 6355 ts_tablet_manager.cc:902] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:02.448635 6338 tablet_copy_client.cc:523] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:02.459971 6338 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:02.469585 6355 tablet_copy_client.cc:308] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:02.470341 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 78edc267e9f44d82b7dbb663628359fe from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:59251)
I20240503 14:16:02.470577 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 78edc267e9f44d82b7dbb663628359fe from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:59251: session id = 7146de39e348442da4a363e95fbe0789-78edc267e9f44d82b7dbb663628359fe
I20240503 14:16:02.473439 2712 tablet_copy_source_session.cc:205] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:02.476560 6355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edc267e9f44d82b7dbb663628359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.531162 2561 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:02.539286 6355 tablet_copy_client.cc:783] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:02.539579 6355 tablet_copy_client.cc:647] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:02.544502 6355 tablet_copy_client.cc:523] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:02.552521 6355 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:02.560153 2430 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.560823 5868 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:16:02.565981 2498 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32927a23842b4725b9dfafbac47fe754. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:02.566720 2691 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.567322 5868 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.655164 6400 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.656159 2691 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.664629 2159 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.665550 6398 ts_tablet_manager.cc:902] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:02.682032 6065 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:02.690429 2430 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:02.698601 6065 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:02.698962 6065 ts_tablet_manager.cc:1367] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.535s user 0.068s sys 0.001s
I20240503 14:16:02.703120 6338 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:02.703596 6338 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:02.703898 6338 ts_tablet_manager.cc:1367] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.244s user 0.050s sys 0.003s
I20240503 14:16:02.704747 6338 raft_consensus.cc:348] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:02.705037 6338 raft_consensus.cc:729] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.708966 6065 raft_consensus.cc:348] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:02.709295 6065 raft_consensus.cc:729] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.709559 6338 consensus_queue.cc:260] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:02.713404 6338 ts_tablet_manager.cc:1398] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240503 14:16:02.714421 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-f139d77b0f0c47809e6408b2771adb73 received from {username='slave'} at 127.20.99.7:57913
I20240503 14:16:02.714550 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-f139d77b0f0c47809e6408b2771adb73 on tablet f139d77b0f0c47809e6408b2771adb73 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:02.722620 2691 consensus_queue.cc:237] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:02.730872 6065 consensus_queue.cc:260] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:02.735590 6065 ts_tablet_manager.cc:1398] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.037s user 0.001s sys 0.003s
I20240503 14:16:02.736300 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-aba6537cdb87400cb8b108b1fbfa6c0f received from {username='slave'} at 127.20.99.6:59085
I20240503 14:16:02.736469 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-aba6537cdb87400cb8b108b1fbfa6c0f on tablet aba6537cdb87400cb8b108b1fbfa6c0f with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:02.739308 2430 consensus_queue.cc:237] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
W20240503 14:16:02.741405 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 832f07a6b12740d08fa67d653b3818bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:02.742395 2432 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.745890 6098 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.769291 2561 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.773357 6098 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.778101 2560 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.778779 6283 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.779119 6355 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:02.779577 6355 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:02.779863 6355 ts_tablet_manager.cc:1367] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.227s user 0.052s sys 0.003s
I20240503 14:16:02.780684 6355 raft_consensus.cc:348] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:02.780920 6355 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.781180 6355 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:02.783439 2301 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:02.784080 6283 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:02.790526 6355 ts_tablet_manager.cc:1398] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
W20240503 14:16:02.791028 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65a196c723884d068b3728e876c87eb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:02.792441 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-78edc267e9f44d82b7dbb663628359fe received from {username='slave'} at 127.20.99.5:59251
I20240503 14:16:02.792634 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 7146de39e348442da4a363e95fbe0789-78edc267e9f44d82b7dbb663628359fe on tablet 78edc267e9f44d82b7dbb663628359fe with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:02.817656 6398 tablet_copy_client.cc:308] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:02.817575 6283 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:02.821619 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet fda4460865854505b5220d7ec3b34c57 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:38575)
I20240503 14:16:02.821851 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet fda4460865854505b5220d7ec3b34c57 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:38575: session id = 285488446ac445c5a5dfe654bfaf9835-fda4460865854505b5220d7ec3b34c57
I20240503 14:16:02.825816 5573 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:02.828696 2712 tablet_copy_source_session.cc:205] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:02.827816 2560 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:02.831128 2432 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:02.834805 6398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4460865854505b5220d7ec3b34c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.842748 2562 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:02.846907 2159 catalog_manager.cc:5484] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:02.867899 2153 catalog_manager.cc:5484] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:02.832191 2301 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:02.876394 6398 tablet_copy_client.cc:783] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:02.876686 6398 tablet_copy_client.cc:647] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:02.879078 6398 tablet_copy_client.cc:523] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:02.891623 4334 raft_consensus.cc:1203] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:02.904966 4203 raft_consensus.cc:1203] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:02.915696 6398 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:03.076560 6065 ts_tablet_manager.cc:902] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:03.077610 2691 consensus_queue.cc:237] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
W20240503 14:16:03.086359 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 832f07a6b12740d08fa67d653b3818bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.087378 2560 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:16:03.087831 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 832f07a6b12740d08fa67d653b3818bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.088238 2430 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:03.089111 6444 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:03.089977 6444 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:03.093326 6444 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.097170 2560 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.104104 2430 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.111593 2153 catalog_manager.cc:5484] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.142138 6065 tablet_copy_client.cc:308] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:03.143040 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 32927a23842b4725b9dfafbac47fe754 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:59085)
I20240503 14:16:03.143314 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 32927a23842b4725b9dfafbac47fe754 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:59085: session id = 285488446ac445c5a5dfe654bfaf9835-32927a23842b4725b9dfafbac47fe754
I20240503 14:16:03.149372 2560 consensus_queue.cc:237] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:03.149777 2582 tablet_copy_source_session.cc:205] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:03.152150 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:16:03.160964 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6325a400641b4d25b95592abb4af7d2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.162159 2430 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:03.162729 6400 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:03.164659 6065 tablet_copy_client.cc:783] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:03.164978 6065 tablet_copy_client.cc:647] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:03.167282 6065 tablet_copy_client.cc:523] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:03.169806 2691 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:03.170444 6400 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:03.172217 6065 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:03.177979 6398 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:03.178455 6398 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:03.178776 6398 ts_tablet_manager.cc:1367] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.263s user 0.059s sys 0.014s
I20240503 14:16:03.181205 6398 raft_consensus.cc:348] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:03.181481 6398 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.182689 6398 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:03.184408 6398 ts_tablet_manager.cc:1398] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:16:03.185475 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-fda4460865854505b5220d7ec3b34c57 received from {username='slave'} at 127.20.99.6:38575
I20240503 14:16:03.185626 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-fda4460865854505b5220d7ec3b34c57 on tablet fda4460865854505b5220d7ec3b34c57 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:03.221954 6400 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.227429 2432 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.235785 2159 catalog_manager.cc:5484] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.244269 4468 raft_consensus.cc:1203] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:03.248121 4334 raft_consensus.cc:1203] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:03.254843 2691 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.278815 2430 consensus_queue.cc:237] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
W20240503 14:16:03.282681 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65a196c723884d068b3728e876c87eb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.283747 2560 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:03.284327 6431 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:03.287863 2301 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:03.288692 6440 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:16:03.291023 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65a196c723884d068b3728e876c87eb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.292892 6283 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.295861 2299 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.302495 2152 catalog_manager.cc:5484] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.312023 2560 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:03.357285 6065 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:03.358994 6065 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:03.359331 6065 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.187s user 0.059s sys 0.003s
I20240503 14:16:03.360113 6065 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:03.360371 6065 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.360617 6065 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:03.363289 6065 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:16:03.368664 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-32927a23842b4725b9dfafbac47fe754 received from {username='slave'} at 127.20.99.6:59085
I20240503 14:16:03.368829 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-32927a23842b4725b9dfafbac47fe754 on tablet 32927a23842b4725b9dfafbac47fe754 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:03.518119 6577 ts_tablet_manager.cc:902] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:03.532135 4334 raft_consensus.cc:1203] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:03.563277 6577 tablet_copy_client.cc:308] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:03.564174 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 832f07a6b12740d08fa67d653b3818bf from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:59251)
I20240503 14:16:03.564370 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 832f07a6b12740d08fa67d653b3818bf from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:59251: session id = 7146de39e348442da4a363e95fbe0789-832f07a6b12740d08fa67d653b3818bf
I20240503 14:16:03.568264 2712 tablet_copy_source_session.cc:205] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:03.582376 2691 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:03.593861 2430 consensus_queue.cc:237] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:03.607923 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:03.619943 6577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832f07a6b12740d08fa67d653b3818bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.624678 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:03.688949 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:03.723222 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:03.730876 4334 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:03.731863 6444 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:03.732115 2562 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:03.733991 6444 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:03.735003 2299 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:03.736016 6444 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:03.748721 6612 ts_tablet_manager.cc:902] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:03.755053 6291 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.761968 4334 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.766404 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:03.768043 2159 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.773710 2560 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:16:03.786022 2628 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fda4460865854505b5220d7ec3b34c57. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.794517 6607 ts_tablet_manager.cc:902] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:03.797380 2299 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.808722 2562 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:03.809335 2691 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:03.809471 6283 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:03.810012 6431 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:16:03.811049 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f631c7713d24f4caf7550bb05147615. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.815438 6616 ts_tablet_manager.cc:902] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:03.818161 6606 ts_tablet_manager.cc:902] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:03.829114 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:03.831727 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:03.857864 6612 tablet_copy_client.cc:308] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:03.858340 6431 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.858832 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 65a196c723884d068b3728e876c87eb2 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:56485)
I20240503 14:16:03.859023 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 65a196c723884d068b3728e876c87eb2 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:56485: session id = 7146de39e348442da4a363e95fbe0789-65a196c723884d068b3728e876c87eb2
I20240503 14:16:03.863979 2452 tablet_copy_source_session.cc:205] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:03.864794 2562 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.865294 2691 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.868402 6612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a196c723884d068b3728e876c87eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.871834 2159 catalog_manager.cc:5484] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index -1 to 4, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:03.899530 6612 tablet_copy_client.cc:783] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:03.908615 6577 tablet_copy_client.cc:783] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:03.909394 6616 tablet_copy_client.cc:308] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:03.912618 6612 tablet_copy_client.cc:647] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:03.912917 6577 tablet_copy_client.cc:647] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:03.915760 6612 tablet_copy_client.cc:523] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:03.920827 6577 tablet_copy_client.cc:523] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:03.925761 6607 tablet_copy_client.cc:308] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:03.926848 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 832f07a6b12740d08fa67d653b3818bf from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:41375)
I20240503 14:16:03.927019 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 832f07a6b12740d08fa67d653b3818bf from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:41375: session id = c2cf5d2244244a2db5ebc7dc46fd430d-832f07a6b12740d08fa67d653b3818bf
I20240503 14:16:03.929234 6612 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:03.929981 2712 tablet_copy_source_session.cc:205] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:03.932462 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 832f07a6b12740d08fa67d653b3818bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.936971 2299 consensus_queue.cc:237] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:03.944036 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 65a196c723884d068b3728e876c87eb2 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:49643)
I20240503 14:16:03.944262 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 65a196c723884d068b3728e876c87eb2 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:49643: session id = d4cbc83202c8473d8c897582d30e5989-65a196c723884d068b3728e876c87eb2
I20240503 14:16:03.948349 6577 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:03.948426 2452 tablet_copy_source_session.cc:205] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:03.950009 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a196c723884d068b3728e876c87eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.957286 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:03.963692 6607 tablet_copy_client.cc:783] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:03.963979 6607 tablet_copy_client.cc:647] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:03.967556 6607 tablet_copy_client.cc:523] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:16:03.968297 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af99824d1853471b867e4ff1f01e9367. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:03.969668 2430 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:03.970484 5961 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:03.976105 6606 tablet_copy_client.cc:308] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:03.977098 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 6325a400641b4d25b95592abb4af7d2d from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:39003)
I20240503 14:16:03.977252 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 6325a400641b4d25b95592abb4af7d2d from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:39003: session id = 7146de39e348442da4a363e95fbe0789-6325a400641b4d25b95592abb4af7d2d
I20240503 14:16:03.980010 6607 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:03.980378 2582 tablet_copy_source_session.cc:205] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:03.981704 6606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6325a400641b4d25b95592abb4af7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.984862 2691 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:03.985657 5961 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:03.988576 6606 tablet_copy_client.cc:783] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:03.993728 6606 tablet_copy_client.cc:647] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:04.003039 6606 tablet_copy_client.cc:523] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:04.004375 6616 tablet_copy_client.cc:783] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:04.004758 6616 tablet_copy_client.cc:647] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:04.007104 6616 tablet_copy_client.cc:523] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:04.017544 6606 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
W20240503 14:16:04.020705 6654 log.cc:924] Time spent T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240503 14:16:04.027720 6615 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.028290 2432 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.033769 2691 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:04.035519 2152 catalog_manager.cc:5484] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.039124 2690 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.055210 6616 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
W20240503 14:16:04.069540 2625 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78edc267e9f44d82b7dbb663628359fe. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.070711 4203 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:04.071444 6444 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:04.075073 2430 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:04.075676 6444 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:04.078019 2299 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:04.078589 6444 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:04.107514 6291 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.108433 4202 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.114799 2301 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.115679 2432 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.120510 2159 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:04.145073 6607 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:04.145546 6607 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:04.145856 6607 ts_tablet_manager.cc:1367] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.166s user 0.048s sys 0.006s
I20240503 14:16:04.146636 6607 raft_consensus.cc:348] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.146903 6607 raft_consensus.cc:729] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.180914 6607 consensus_queue.cc:260] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.198669 6607 ts_tablet_manager.cc:1398] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20240503 14:16:04.200734 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-832f07a6b12740d08fa67d653b3818bf received from {username='slave'} at 127.20.99.8:41375
I20240503 14:16:04.200901 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-832f07a6b12740d08fa67d653b3818bf on tablet 832f07a6b12740d08fa67d653b3818bf with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:04.202401 6612 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:04.209146 6612 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:04.211491 6612 ts_tablet_manager.cc:1367] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.282s user 0.066s sys 0.000s
I20240503 14:16:04.212400 6612 raft_consensus.cc:348] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.226039 6612 raft_consensus.cc:729] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.226447 6612 consensus_queue.cc:260] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.231426 6612 ts_tablet_manager.cc:1398] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 14:16:04.232373 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-65a196c723884d068b3728e876c87eb2 received from {username='slave'} at 127.20.99.5:56485
I20240503 14:16:04.232524 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 7146de39e348442da4a363e95fbe0789-65a196c723884d068b3728e876c87eb2 on tablet 65a196c723884d068b3728e876c87eb2 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:04.232656 6577 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:04.237543 6577 tablet_bootstrap.cc:492] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:04.238334 6577 ts_tablet_manager.cc:1367] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.290s user 0.063s sys 0.004s
I20240503 14:16:04.238986 4202 raft_consensus.cc:1203] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:04.239563 6577 raft_consensus.cc:348] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.239840 6577 raft_consensus.cc:729] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.240126 6577 consensus_queue.cc:260] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.240985 6577 ts_tablet_manager.cc:1398] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:04.241789 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-832f07a6b12740d08fa67d653b3818bf received from {username='slave'} at 127.20.99.5:59251
I20240503 14:16:04.241943 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 7146de39e348442da4a363e95fbe0789-832f07a6b12740d08fa67d653b3818bf on tablet 832f07a6b12740d08fa67d653b3818bf with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:04.259583 6606 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:04.260166 6606 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:04.260521 6606 ts_tablet_manager.cc:1367] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.243s user 0.053s sys 0.006s
I20240503 14:16:04.261575 6606 raft_consensus.cc:348] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.261894 6606 raft_consensus.cc:729] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.274822 6607 ts_tablet_manager.cc:902] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:04.283576 6606 consensus_queue.cc:260] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.292811 6606 ts_tablet_manager.cc:1398] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20240503 14:16:04.293876 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-6325a400641b4d25b95592abb4af7d2d received from {username='slave'} at 127.20.99.5:39003
I20240503 14:16:04.294076 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 7146de39e348442da4a363e95fbe0789-6325a400641b4d25b95592abb4af7d2d on tablet 6325a400641b4d25b95592abb4af7d2d with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:04.296970 4202 raft_consensus.cc:1203] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:04.316653 2432 consensus_queue.cc:237] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
W20240503 14:16:04.320806 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f631c7713d24f4caf7550bb05147615. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:04.321027 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f631c7713d24f4caf7550bb05147615. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.321951 2691 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:04.322546 6431 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:04.337848 6607 tablet_copy_client.cc:308] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:04.338601 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet fda4460865854505b5220d7ec3b34c57 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:41375)
I20240503 14:16:04.338819 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet fda4460865854505b5220d7ec3b34c57 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:41375: session id = c2cf5d2244244a2db5ebc7dc46fd430d-fda4460865854505b5220d7ec3b34c57
I20240503 14:16:04.339502 2562 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:04.340297 6730 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:04.342473 2712 tablet_copy_source_session.cc:205] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:04.345682 6283 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.346395 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4460865854505b5220d7ec3b34c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:04.350682 2691 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.355674 2152 catalog_manager.cc:5484] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.359745 2562 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.365643 2432 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:04.403561 6607 tablet_copy_client.cc:783] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:04.403975 6607 tablet_copy_client.cc:647] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
W20240503 14:16:04.404635 2368 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b359fb8d9afc4c9c856897a722100304. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.405395 2562 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:04.406526 2301 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:04.407265 6283 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:04.407788 6283 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:04.408118 6607 tablet_copy_client.cc:523] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:04.416062 6730 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.418467 6607 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:04.420153 2301 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.428355 2152 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.430871 2562 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:04.503579 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:04.525753 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.543752 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:04.547209 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.568764 6616 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:04.569234 6616 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:04.569558 6616 ts_tablet_manager.cc:1367] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.514s user 0.058s sys 0.005s
I20240503 14:16:04.581473 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:04.584857 6616 raft_consensus.cc:348] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.585176 6616 raft_consensus.cc:729] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.587495 6616 consensus_queue.cc:260] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.613765 4599 raft_consensus.cc:1203] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:04.629170 6789 ts_tablet_manager.cc:902] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:16:04.643508 6616 ts_tablet_manager.cc:1398] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.074s user 0.007s sys 0.003s
I20240503 14:16:04.644326 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-65a196c723884d068b3728e876c87eb2 received from {username='slave'} at 127.20.99.7:49643
I20240503 14:16:04.644479 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-65a196c723884d068b3728e876c87eb2 on tablet 65a196c723884d068b3728e876c87eb2 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:04.645581 6793 ts_tablet_manager.cc:902] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:04.666429 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.671468 4467 raft_consensus.cc:1203] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:04.721530 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:04.727074 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.760239 6819 ts_tablet_manager.cc:902] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:04.793702 6823 ts_tablet_manager.cc:902] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:04.794461 6789 tablet_copy_client.cc:308] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:16:04.795130 2320 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet af99824d1853471b867e4ff1f01e9367 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:43323)
I20240503 14:16:04.795364 2320 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet af99824d1853471b867e4ff1f01e9367 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:43323: session id = d4cbc83202c8473d8c897582d30e5989-af99824d1853471b867e4ff1f01e9367
I20240503 14:16:04.798321 2320 tablet_copy_source_session.cc:205] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:04.807268 2562 consensus_queue.cc:237] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:04.807559 6819 tablet_copy_client.cc:308] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:04.808362 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 5f631c7713d24f4caf7550bb05147615 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:52553)
I20240503 14:16:04.808542 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 5f631c7713d24f4caf7550bb05147615 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:52553: session id = 285488446ac445c5a5dfe654bfaf9835-5f631c7713d24f4caf7550bb05147615
I20240503 14:16:04.813791 2452 tablet_copy_source_session.cc:205] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:04.815984 6819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f631c7713d24f4caf7550bb05147615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:04.821419 6789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af99824d1853471b867e4ff1f01e9367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:04.837235 6607 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:04.838099 6819 tablet_copy_client.cc:783] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:04.838438 6819 tablet_copy_client.cc:647] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:04.840023 6607 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:04.840366 6607 ts_tablet_manager.cc:1367] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.424s user 0.061s sys 0.016s
I20240503 14:16:04.840698 6819 tablet_copy_client.cc:523] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:04.841624 6607 raft_consensus.cc:348] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:04.842278 6607 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.842628 6607 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:04.843403 6607 ts_tablet_manager.cc:1398] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:04.844568 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-fda4460865854505b5220d7ec3b34c57 received from {username='slave'} at 127.20.99.8:41375
I20240503 14:16:04.844715 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-fda4460865854505b5220d7ec3b34c57 on tablet fda4460865854505b5220d7ec3b34c57 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:04.849288 6607 ts_tablet_manager.cc:902] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:04.850629 6819 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:04.860965 4334 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:04.861572 6400 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:04.866837 4599 raft_consensus.cc:1203] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:16:04.869946 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aba6537cdb87400cb8b108b1fbfa6c0f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:04.870102 2432 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:04.870749 6400 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:04.878352 6607 tablet_copy_client.cc:308] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:04.879427 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 5f631c7713d24f4caf7550bb05147615 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46595)
I20240503 14:16:04.879639 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 5f631c7713d24f4caf7550bb05147615 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46595: session id = c2cf5d2244244a2db5ebc7dc46fd430d-5f631c7713d24f4caf7550bb05147615
I20240503 14:16:04.884392 4467 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:04.885138 6400 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:04.886059 2452 tablet_copy_source_session.cc:205] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:04.890539 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f631c7713d24f4caf7550bb05147615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:04.896467 2301 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:04.897325 6400 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:04.898047 6607 tablet_copy_client.cc:783] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:04.898279 6607 tablet_copy_client.cc:647] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:04.900928 4202 raft_consensus.cc:1203] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:04.912853 6607 tablet_copy_client.cc:523] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:04.917341 6607 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:04.935454 6793 tablet_copy_client.cc:308] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:04.936772 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 78edc267e9f44d82b7dbb663628359fe from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:57913)
I20240503 14:16:04.936956 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 78edc267e9f44d82b7dbb663628359fe from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:57913: session id = d4cbc83202c8473d8c897582d30e5989-78edc267e9f44d82b7dbb663628359fe
I20240503 14:16:04.942453 2712 tablet_copy_source_session.cc:205] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:04.944733 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edc267e9f44d82b7dbb663628359fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:16:04.954532 2497 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 52078.2us
I20240503 14:16:04.960657 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.964799 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:04.964922 4335 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:04.973987 2301 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:04.979979 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.982018 6823 tablet_copy_client.cc:308] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:04.982700 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet b359fb8d9afc4c9c856897a722100304 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:52553)
I20240503 14:16:04.982941 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet b359fb8d9afc4c9c856897a722100304 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:52553: session id = 285488446ac445c5a5dfe654bfaf9835-b359fb8d9afc4c9c856897a722100304
I20240503 14:16:04.987543 2452 tablet_copy_source_session.cc:205] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:04.995406 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:04.964530 6857 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.017241 6823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b359fb8d9afc4c9c856897a722100304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.024411 2159 catalog_manager.cc:5484] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:04.971057 6789 tablet_copy_client.cc:783] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.042521 6789 tablet_copy_client.cc:647] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.042749 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:05.049072 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:05.052616 6789 tablet_copy_client.cc:523] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.064275 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:05.073168 4467 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.074304 6823 tablet_copy_client.cc:783] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.074566 6823 tablet_copy_client.cc:647] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.077219 2432 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.096246 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:05.096668 6793 tablet_copy_client.cc:783] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.097225 6793 tablet_copy_client.cc:647] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.098127 6789 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:05.101055 6793 tablet_copy_client.cc:523] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.122879 6793 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:05.147972 6607 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.148469 6607 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:05.148759 6607 ts_tablet_manager.cc:1367] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.232s user 0.051s sys 0.006s
I20240503 14:16:05.149068 2691 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:05.149601 6607 raft_consensus.cc:348] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.149891 6607 raft_consensus.cc:729] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.163029 6607 consensus_queue.cc:260] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.163970 6607 ts_tablet_manager.cc:1398] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240503 14:16:05.164901 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-5f631c7713d24f4caf7550bb05147615 received from {username='slave'} at 127.20.99.8:46595
I20240503 14:16:05.165055 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-5f631c7713d24f4caf7550bb05147615 on tablet 5f631c7713d24f4caf7550bb05147615 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:05.167233 2301 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:05.167857 6291 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:05.168413 2628 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78edc267e9f44d82b7dbb663628359fe. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:05.170960 2625 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.171496 6823 tablet_copy_client.cc:523] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.171880 4202 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:05.172866 6291 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:05.175606 6823 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:05.179811 2432 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:05.180619 6291 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:05.312281 6291 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:05.316067 2431 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:05.318957 2301 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:05.325070 2153 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:05.326254 4202 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:05.381356 4599 raft_consensus.cc:1203] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:05.388171 6910 ts_tablet_manager.cc:902] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:05.399116 6789 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.399701 6789 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:05.400074 6789 ts_tablet_manager.cc:1367] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.302s user 0.045s sys 0.009s
I20240503 14:16:05.402061 6789 raft_consensus.cc:348] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:05.402352 6789 raft_consensus.cc:729] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.404533 6789 consensus_queue.cc:260] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:05.410614 6789 ts_tablet_manager.cc:1398] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240503 14:16:05.414569 2320 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-af99824d1853471b867e4ff1f01e9367 received from {username='slave'} at 127.20.99.7:43323
I20240503 14:16:05.414767 2320 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-af99824d1853471b867e4ff1f01e9367 on tablet af99824d1853471b867e4ff1f01e9367 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:05.415961 2432 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:05.419435 2562 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:16:05.419950 2365 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b359fb8d9afc4c9c856897a722100304. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:05.420116 2368 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.420603 6730 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:05.431440 6910 tablet_copy_client.cc:308] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:05.432288 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet aba6537cdb87400cb8b108b1fbfa6c0f from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:39003)
I20240503 14:16:05.432475 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet aba6537cdb87400cb8b108b1fbfa6c0f from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:39003: session id = 7146de39e348442da4a363e95fbe0789-aba6537cdb87400cb8b108b1fbfa6c0f
I20240503 14:16:05.436625 2301 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:05.438067 6730 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:05.440655 2582 tablet_copy_source_session.cc:205] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.442268 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aba6537cdb87400cb8b108b1fbfa6c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.457067 6819 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.457700 6819 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:05.458089 6819 ts_tablet_manager.cc:1367] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.608s user 0.054s sys 0.007s
I20240503 14:16:05.466936 6920 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:05.474220 6793 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.474700 6793 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:05.475003 6793 ts_tablet_manager.cc:1367] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.352s user 0.050s sys 0.006s
I20240503 14:16:05.476856 2299 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:05.483136 2152 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:05.489400 6910 tablet_copy_client.cc:783] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.489730 6910 tablet_copy_client.cc:647] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.490501 6793 raft_consensus.cc:348] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:05.490784 6793 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.491051 6793 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:05.491662 6793 ts_tablet_manager.cc:1398] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 14:16:05.494047 6819 raft_consensus.cc:348] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.494302 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-78edc267e9f44d82b7dbb663628359fe received from {username='slave'} at 127.20.99.7:57913
I20240503 14:16:05.494410 6819 raft_consensus.cc:729] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.494442 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-78edc267e9f44d82b7dbb663628359fe on tablet 78edc267e9f44d82b7dbb663628359fe with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:05.497119 6819 consensus_queue.cc:260] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.504834 2562 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:05.506939 6819 ts_tablet_manager.cc:1398] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20240503 14:16:05.512019 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-5f631c7713d24f4caf7550bb05147615 received from {username='slave'} at 127.20.99.6:52553
I20240503 14:16:05.512181 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-5f631c7713d24f4caf7550bb05147615 on tablet 5f631c7713d24f4caf7550bb05147615 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:05.516100 6910 tablet_copy_client.cc:523] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.522743 6910 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:05.545706 2301 consensus_queue.cc:237] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.559114 4467 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:05.561045 6615 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:16:05.573462 2237 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af99824d1853471b867e4ff1f01e9367. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.573714 6823 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.574194 6823 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:05.575310 2691 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:05.575748 6823 ts_tablet_manager.cc:1367] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.400s user 0.063s sys 0.000s
I20240503 14:16:05.575861 6615 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:05.576610 6823 raft_consensus.cc:348] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.576853 6823 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.577113 6823 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:05.581686 6823 ts_tablet_manager.cc:1398] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:16:05.585502 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-b359fb8d9afc4c9c856897a722100304 received from {username='slave'} at 127.20.99.6:52553
I20240503 14:16:05.585752 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-b359fb8d9afc4c9c856897a722100304 on tablet b359fb8d9afc4c9c856897a722100304 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:05.588325 2432 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:05.589056 6615 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:05.599603 2432 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:05.606060 2691 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:05.606184 2152 catalog_manager.cc:5484] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:05.596709 6615 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:05.633216 4469 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:05.585533 4467 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:05.701681 4335 raft_consensus.cc:1203] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:16:05.702471 2234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:45269, user_credentials={real_user=slave}} blocked reactor thread for 58153.9us
I20240503 14:16:05.719033 6972 ts_tablet_manager.cc:902] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:05.719302 6910 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.720134 6910 tablet_bootstrap.cc:492] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:05.720494 6910 ts_tablet_manager.cc:1367] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.198s user 0.062s sys 0.006s
I20240503 14:16:05.721520 6910 raft_consensus.cc:348] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:05.721859 6910 raft_consensus.cc:729] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.722458 6910 consensus_queue.cc:260] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:05.724501 6910 ts_tablet_manager.cc:1398] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:16:05.725315 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-aba6537cdb87400cb8b108b1fbfa6c0f received from {username='slave'} at 127.20.99.5:39003
I20240503 14:16:05.725472 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 7146de39e348442da4a363e95fbe0789-aba6537cdb87400cb8b108b1fbfa6c0f on tablet aba6537cdb87400cb8b108b1fbfa6c0f with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:05.786782 2562 consensus_queue.cc:237] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:05.790019 2691 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:05.790585 6857 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:05.792527 2301 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:16:05.793160 2497 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 059a4a35aa94412994bb431342ba0753. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.793692 6972 tablet_copy_client.cc:308] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:05.794857 6995 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:05.795432 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 78edc267e9f44d82b7dbb663628359fe from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:41375)
I20240503 14:16:05.795629 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 78edc267e9f44d82b7dbb663628359fe from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:41375: session id = c2cf5d2244244a2db5ebc7dc46fd430d-78edc267e9f44d82b7dbb663628359fe
I20240503 14:16:05.799162 2712 tablet_copy_source_session.cc:205] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.802537 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78edc267e9f44d82b7dbb663628359fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.813817 6972 tablet_copy_client.cc:783] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.813795 2299 consensus_queue.cc:237] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:05.813795 6857 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.814476 6972 tablet_copy_client.cc:647] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.816709 6972 tablet_copy_client.cc:523] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.819425 4469 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:05.819803 2691 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:05.821012 2300 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.821831 6972 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:05.824575 6959 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240503 14:16:05.825268 2237 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af99824d1853471b867e4ff1f01e9367. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.825459 6615 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:05.826308 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af99824d1853471b867e4ff1f01e9367. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:05.837661 2691 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.848085 2152 catalog_manager.cc:5484] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.856866 6793 ts_tablet_manager.cc:902] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:05.862834 4202 raft_consensus.cc:1203] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:16:05.891481 6793 tablet_copy_client.cc:308] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:05.909400 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet b359fb8d9afc4c9c856897a722100304 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:49643)
I20240503 14:16:05.909593 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet b359fb8d9afc4c9c856897a722100304 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:49643: session id = d4cbc83202c8473d8c897582d30e5989-b359fb8d9afc4c9c856897a722100304
I20240503 14:16:05.914461 2452 tablet_copy_source_session.cc:205] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:05.917029 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b359fb8d9afc4c9c856897a722100304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.917655 4335 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:05.928418 2430 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:05.929533 6818 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:05.933341 6818 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.935295 2430 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.951541 4469 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.952451 2691 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.958062 2152 catalog_manager.cc:5484] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:05.970836 6972 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:05.971343 6972 tablet_bootstrap.cc:492] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:05.971665 6972 ts_tablet_manager.cc:1367] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.150s user 0.056s sys 0.025s
I20240503 14:16:05.972561 6972 raft_consensus.cc:348] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:05.972846 6972 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.973137 6972 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:05.984058 6793 tablet_copy_client.cc:783] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:05.984439 6793 tablet_copy_client.cc:647] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:05.987983 6793 tablet_copy_client.cc:523] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:05.989051 2562 consensus_queue.cc:237] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:05.992209 6793 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:06.000873 2691 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.001724 6857 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:06.004981 6972 ts_tablet_manager.cc:1398] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.033s user 0.006s sys 0.002s
I20240503 14:16:06.005594 4202 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.006244 6995 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:06.007069 2430 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.010003 6995 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:06.050959 6995 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.056924 4203 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.066854 2152 catalog_manager.cc:5484] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 reported cstate change: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.084332 2430 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.105495 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-78edc267e9f44d82b7dbb663628359fe received from {username='slave'} at 127.20.99.8:41375
I20240503 14:16:06.105154 2691 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.105649 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-78edc267e9f44d82b7dbb663628359fe on tablet 78edc267e9f44d82b7dbb663628359fe with peer c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:06.140750 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6325a400641b4d25b95592abb4af7d2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:06.213263 6910 ts_tablet_manager.cc:902] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:06.216882 2562 consensus_queue.cc:237] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:06.224260 2691 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:16:06.224812 2498 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 059a4a35aa94412994bb431342ba0753. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:06.225586 6995 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:16:06.226821 2497 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 059a4a35aa94412994bb431342ba0753. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:06.227095 6793 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:06.229533 6793 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:06.229517 2301 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.229878 6793 ts_tablet_manager.cc:1367] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.238s user 0.057s sys 0.008s
I20240503 14:16:06.231073 6857 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:06.233609 6793 raft_consensus.cc:348] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:06.233865 6793 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.233922 6857 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.234125 6793 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:06.234763 6793 ts_tablet_manager.cc:1398] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:16:06.241564 2301 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.248888 2152 catalog_manager.cc:5484] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.253461 2430 consensus_queue.cc:237] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:06.264037 2691 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:06.271274 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-b359fb8d9afc4c9c856897a722100304 received from {username='slave'} at 127.20.99.7:49643
I20240503 14:16:06.271416 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-b359fb8d9afc4c9c856897a722100304 on tablet b359fb8d9afc4c9c856897a722100304 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:06.272889 7100 ts_tablet_manager.cc:902] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
W20240503 14:16:06.292104 2368 consensus_peers.cc:485] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9a474791f9b4585aa0114162705c96f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:06.293344 2301 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:06.294286 2562 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:16:06.295140 6730 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:06.297739 6730 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:06.302739 6910 tablet_copy_client.cc:308] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:06.303514 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 059a4a35aa94412994bb431342ba0753 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:39003)
I20240503 14:16:06.303689 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 059a4a35aa94412994bb431342ba0753 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:39003: session id = 7146de39e348442da4a363e95fbe0789-059a4a35aa94412994bb431342ba0753
I20240503 14:16:06.306785 2582 tablet_copy_source_session.cc:205] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.308878 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a4a35aa94412994bb431342ba0753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.311430 4599 raft_consensus.cc:1203] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:16:06.340330 6910 tablet_copy_client.cc:783] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.341086 6910 tablet_copy_client.cc:647] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.341781 6730 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:06.343401 6910 tablet_copy_client.cc:523] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.349159 6910 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:06.360168 7100 tablet_copy_client.cc:308] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:16:06.361372 2320 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet af99824d1853471b867e4ff1f01e9367 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:48475)
I20240503 14:16:06.361560 2320 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet af99824d1853471b867e4ff1f01e9367 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:48475: session id = 285488446ac445c5a5dfe654bfaf9835-af99824d1853471b867e4ff1f01e9367
I20240503 14:16:06.364704 2320 tablet_copy_source_session.cc:205] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.366302 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af99824d1853471b867e4ff1f01e9367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.377107 7100 tablet_copy_client.cc:783] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.377517 7100 tablet_copy_client.cc:647] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.379343 7100 tablet_copy_client.cc:523] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.385097 7100 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:06.426376 2562 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:06.433949 2152 catalog_manager.cc:5484] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:06.438807 6972 ts_tablet_manager.cc:902] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:06.440868 2430 consensus_queue.cc:237] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:06.453971 4469 raft_consensus.cc:1203] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:06.453817 2301 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:06.465655 7153 ts_tablet_manager.cc:902] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:16:06.477511 4599 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.478940 7124 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:16:06.486760 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc88c5dc94844f1c8400c31ad1165a32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:06.501055 6910 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:06.501674 6910 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:06.502048 6910 ts_tablet_manager.cc:1367] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.153s user 0.061s sys 0.004s
I20240503 14:16:06.503132 6910 raft_consensus.cc:348] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:06.503528 6910 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.503880 6910 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:06.504698 6910 ts_tablet_manager.cc:1398] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:16:06.505751 7153 tablet_copy_client.cc:308] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:16:06.506470 2320 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet af99824d1853471b867e4ff1f01e9367 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:56357)
I20240503 14:16:06.506686 2320 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet af99824d1853471b867e4ff1f01e9367 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:56357: session id = 7146de39e348442da4a363e95fbe0789-af99824d1853471b867e4ff1f01e9367
I20240503 14:16:06.510119 2320 tablet_copy_source_session.cc:205] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.511639 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af99824d1853471b867e4ff1f01e9367. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.516098 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-059a4a35aa94412994bb431342ba0753 received from {username='slave'} at 127.20.99.5:39003
I20240503 14:16:06.516292 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 7146de39e348442da4a363e95fbe0789-059a4a35aa94412994bb431342ba0753 on tablet 059a4a35aa94412994bb431342ba0753 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:06.522648 2691 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.524086 2562 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:06.529359 6920 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:06.532622 6920 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:06.534622 7153 tablet_copy_client.cc:783] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.534968 7153 tablet_copy_client.cc:647] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.537901 7153 tablet_copy_client.cc:523] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.543692 7153 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:06.549602 6972 tablet_copy_client.cc:308] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:06.560866 6920 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:06.573495 2159 catalog_manager.cc:5484] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:06.583968 2691 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:06.584517 2562 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:06.612860 7153 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:06.613464 7153 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:06.613816 7153 ts_tablet_manager.cc:1367] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.070s user 0.042s sys 0.021s
I20240503 14:16:06.614732 7153 raft_consensus.cc:348] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:06.615132 7153 raft_consensus.cc:729] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.615480 7153 consensus_queue.cc:260] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:06.616168 7153 ts_tablet_manager.cc:1398] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:06.617077 2320 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-af99824d1853471b867e4ff1f01e9367 received from {username='slave'} at 127.20.99.5:56357
I20240503 14:16:06.617249 2320 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session 7146de39e348442da4a363e95fbe0789-af99824d1853471b867e4ff1f01e9367 on tablet af99824d1853471b867e4ff1f01e9367 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:06.633431 4599 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:06.653223 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 6325a400641b4d25b95592abb4af7d2d from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:48453)
I20240503 14:16:06.653445 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 6325a400641b4d25b95592abb4af7d2d from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:48453: session id = c2cf5d2244244a2db5ebc7dc46fd430d-6325a400641b4d25b95592abb4af7d2d
I20240503 14:16:06.656315 2582 tablet_copy_source_session.cc:205] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.657742 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6325a400641b4d25b95592abb4af7d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.666410 6972 tablet_copy_client.cc:783] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.666677 6972 tablet_copy_client.cc:647] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.669106 6972 tablet_copy_client.cc:523] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.689268 6972 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:06.733731 7197 ts_tablet_manager.cc:902] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:06.769629 7153 tablet_replica.cc:330] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: stopping tablet replica
I20240503 14:16:06.770299 7153 raft_consensus.cc:2229] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:06.770681 7153 raft_consensus.cc:2258] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:06.772490 7153 ts_tablet_manager.cc:1875] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:06.787499 7153 ts_tablet_manager.cc:1888] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:06.787725 7153 log.cc:1196] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-4/wal/wals/059a4a35aa94412994bb431342ba0753
I20240503 14:16:06.788301 7153 ts_tablet_manager.cc:902] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:06.807642 7197 tablet_copy_client.cc:308] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:06.809111 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 059a4a35aa94412994bb431342ba0753 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:48453)
I20240503 14:16:06.809283 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 059a4a35aa94412994bb431342ba0753 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:48453: session id = c2cf5d2244244a2db5ebc7dc46fd430d-059a4a35aa94412994bb431342ba0753
I20240503 14:16:06.812252 2582 tablet_copy_source_session.cc:205] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.818800 7197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a4a35aa94412994bb431342ba0753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.834761 4202 raft_consensus.cc:1203] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:16:06.842916 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:06.854911 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:06.869448 7153 tablet_copy_client.cc:308] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:06.870421 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 059a4a35aa94412994bb431342ba0753 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:39003)
I20240503 14:16:06.870610 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 059a4a35aa94412994bb431342ba0753 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:39003: session id = 7146de39e348442da4a363e95fbe0789-059a4a35aa94412994bb431342ba0753
I20240503 14:16:06.873778 2582 tablet_copy_source_session.cc:205] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:06.877763 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:06.881654 7153 ts_tablet_manager.cc:1875] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:06.882591 7197 tablet_copy_client.cc:783] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:06.882907 7197 tablet_copy_client.cc:647] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:06.886734 7197 tablet_copy_client.cc:523] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:06.891165 7197 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:06.925001 6972 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:06.925537 6972 tablet_bootstrap.cc:492] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:06.925806 6972 ts_tablet_manager.cc:1367] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.237s user 0.055s sys 0.006s
I20240503 14:16:06.958580 7153 ts_tablet_manager.cc:1888] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:16:06.962699 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:06.962695 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:06.964104 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:06.973485 6972 raft_consensus.cc:348] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:06.973775 6972 raft_consensus.cc:729] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.959808 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059a4a35aa94412994bb431342ba0753. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.982563 6972 consensus_queue.cc:260] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:06.983743 6972 ts_tablet_manager.cc:1398] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.058s user 0.003s sys 0.000s
I20240503 14:16:06.984426 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-6325a400641b4d25b95592abb4af7d2d received from {username='slave'} at 127.20.99.8:48453
I20240503 14:16:06.984555 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-6325a400641b4d25b95592abb4af7d2d on tablet 6325a400641b4d25b95592abb4af7d2d with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:06.997897 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:07.010331 4599 raft_consensus.cc:1203] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:07.015326 7153 tablet_copy_client.cc:783] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:07.015584 7153 tablet_copy_client.cc:647] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:07.018849 7243 ts_tablet_manager.cc:902] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:07.023087 7153 tablet_copy_client.cc:523] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:07.027667 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.030261 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:07.031200 7244 ts_tablet_manager.cc:902] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:07.045670 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.064380 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.064996 7243 tablet_copy_client.cc:308] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:07.065223 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.067910 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet dc88c5dc94844f1c8400c31ad1165a32 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:49643)
I20240503 14:16:07.068056 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet dc88c5dc94844f1c8400c31ad1165a32 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:49643: session id = d4cbc83202c8473d8c897582d30e5989-dc88c5dc94844f1c8400c31ad1165a32
I20240503 14:16:07.071103 2452 tablet_copy_source_session.cc:205] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:07.076035 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc88c5dc94844f1c8400c31ad1165a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:07.104367 7153 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:07.125200 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:07.135790 2430 consensus_queue.cc:237] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:07.140408 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:07.152082 7243 tablet_copy_client.cc:783] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:07.153892 7243 tablet_copy_client.cc:647] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
W20240503 14:16:07.165225 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f631c7713d24f4caf7550bb05147615. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:07.168778 4599 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.169216 7243 tablet_copy_client.cc:523] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:07.170140 7133 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.172799 2691 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.173270 7133 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.176164 2562 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.177311 7133 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.189536 7243 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:07.197621 4333 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.199144 7124 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.241312 7244 tablet_copy_client.cc:308] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:07.258792 7243 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.259337 7243 tablet_bootstrap.cc:492] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:07.259639 7243 ts_tablet_manager.cc:1367] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.070s user 0.041s sys 0.017s
I20240503 14:16:07.260509 7243 raft_consensus.cc:348] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:07.260783 7243 raft_consensus.cc:729] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:07.261052 7243 consensus_queue.cc:260] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:07.261884 7197 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.262346 7197 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:07.262652 7197 ts_tablet_manager.cc:1367] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.372s user 0.051s sys 0.007s
I20240503 14:16:07.266222 2560 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:07.266113 7133 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:07.268466 7243 ts_tablet_manager.cc:1398] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20240503 14:16:07.273602 2159 catalog_manager.cc:5484] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:07.281338 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-dc88c5dc94844f1c8400c31ad1165a32 received from {username='slave'} at 127.20.99.7:49643
I20240503 14:16:07.281499 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-dc88c5dc94844f1c8400c31ad1165a32 on tablet dc88c5dc94844f1c8400c31ad1165a32 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:07.291473 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:07.293342 7197 raft_consensus.cc:348] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:07.293632 7197 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:07.293870 7197 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:07.297402 7197 ts_tablet_manager.cc:1398] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20240503 14:16:07.298074 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-059a4a35aa94412994bb431342ba0753 received from {username='slave'} at 127.20.99.8:48453
I20240503 14:16:07.298192 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-059a4a35aa94412994bb431342ba0753 on tablet 059a4a35aa94412994bb431342ba0753 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:07.307835 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:16:07.309166 2368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 110473us
I20240503 14:16:07.347138 4335 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:07.352542 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet a9a474791f9b4585aa0114162705c96f from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:52553)
I20240503 14:16:07.352770 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet a9a474791f9b4585aa0114162705c96f from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:52553: session id = 285488446ac445c5a5dfe654bfaf9835-a9a474791f9b4585aa0114162705c96f
I20240503 14:16:07.356180 2452 tablet_copy_source_session.cc:205] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:07.381826 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9a474791f9b4585aa0114162705c96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:07.388128 2691 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:07.388823 4598 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:07.396361 7244 tablet_copy_client.cc:783] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:07.396626 7244 tablet_copy_client.cc:647] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:07.399837 7244 tablet_copy_client.cc:523] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:07.405725 4469 raft_consensus.cc:1203] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:07.418031 7244 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
I20240503 14:16:07.424398 7100 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.424882 7100 tablet_bootstrap.cc:492] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:07.425170 7100 ts_tablet_manager.cc:1367] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 1.040s user 0.075s sys 0.024s
I20240503 14:16:07.432241 7100 raft_consensus.cc:348] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:07.435792 7100 raft_consensus.cc:729] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:07.436107 7100 consensus_queue.cc:260] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:07.437012 7100 ts_tablet_manager.cc:1398] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:16:07.437867 2320 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-af99824d1853471b867e4ff1f01e9367 received from {username='slave'} at 127.20.99.6:48475
I20240503 14:16:07.438022 2320 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-af99824d1853471b867e4ff1f01e9367 on tablet af99824d1853471b867e4ff1f01e9367 with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:07.531428 7153 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.531912 7153 tablet_bootstrap.cc:492] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:07.532186 7153 ts_tablet_manager.cc:1367] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.428s user 0.053s sys 0.007s
I20240503 14:16:07.534734 7153 raft_consensus.cc:348] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:07.534986 7153 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:07.535908 7153 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:07.543870 7153 ts_tablet_manager.cc:1398] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20240503 14:16:07.544917 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-059a4a35aa94412994bb431342ba0753 received from {username='slave'} at 127.20.99.5:39003
I20240503 14:16:07.545064 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 7146de39e348442da4a363e95fbe0789-059a4a35aa94412994bb431342ba0753 on tablet 059a4a35aa94412994bb431342ba0753 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:07.557353 4598 raft_consensus.cc:1203] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:07.574762 4335 raft_consensus.cc:1203] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:16:07.580344 7244 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:07.580861 7244 tablet_bootstrap.cc:492] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:07.581090 7244 ts_tablet_manager.cc:1367] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.163s user 0.063s sys 0.002s
I20240503 14:16:07.581792 7244 raft_consensus.cc:348] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:07.581987 7244 raft_consensus.cc:729] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:07.582224 7244 consensus_queue.cc:260] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:07.582855 7244 ts_tablet_manager.cc:1398] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:16:07.583640 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-a9a474791f9b4585aa0114162705c96f received from {username='slave'} at 127.20.99.6:52553
I20240503 14:16:07.583765 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-a9a474791f9b4585aa0114162705c96f on tablet a9a474791f9b4585aa0114162705c96f with peer 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:07.667841 7153 ts_tablet_manager.cc:902] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:07.672420 7153 tablet_copy_client.cc:308] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:07.673219 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 5f631c7713d24f4caf7550bb05147615 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:56485)
I20240503 14:16:07.673393 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 5f631c7713d24f4caf7550bb05147615 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:56485: session id = 7146de39e348442da4a363e95fbe0789-5f631c7713d24f4caf7550bb05147615
I20240503 14:16:07.681236 2430 consensus_queue.cc:237] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:07.683009 2452 tablet_copy_source_session.cc:205] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:07.685539 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f631c7713d24f4caf7550bb05147615. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:07.695513 7153 tablet_copy_client.cc:783] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:07.695838 7153 tablet_copy_client.cc:647] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:07.699941 4202 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.700630 7124 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:07.703006 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65a196c723884d068b3728e876c87eb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:07.708158 2560 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.708958 7133 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.720368 7153 tablet_copy_client.cc:523] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:07.722386 4469 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.723029 7133 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.736680 2299 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:07.737459 7133 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:07.749082 4203 raft_consensus.cc:1203] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:07.749626 7133 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:07.750337 7153 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:07.757280 4203 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:07.765194 2153 catalog_manager.cc:5484] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 reported cstate change: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:07.764938 2560 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:07.773912 4469 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:07.784608 2299 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:07.804529 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:07.804738 4335 raft_consensus.cc:1203] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:16:07.815622 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.870725 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:07.873179 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:07.876242 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.909665 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:07.964702 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:07.972860 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:08.245040 7153 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:08.245767 7153 tablet_bootstrap.cc:492] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:08.246080 7153 ts_tablet_manager.cc:1367] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.496s user 0.069s sys 0.001s
I20240503 14:16:08.246976 7153 raft_consensus.cc:348] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:08.247277 7153 raft_consensus.cc:729] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:08.247658 7153 consensus_queue.cc:260] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:08.248517 7153 ts_tablet_manager.cc:1398] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:16:08.252509 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-5f631c7713d24f4caf7550bb05147615 received from {username='slave'} at 127.20.99.5:56485
I20240503 14:16:08.252700 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 7146de39e348442da4a363e95fbe0789-5f631c7713d24f4caf7550bb05147615 on tablet 5f631c7713d24f4caf7550bb05147615 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:08.277105 4203 raft_consensus.cc:1203] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:16:08.286707 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:08.292194 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:08.311903 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:08.325030 7449 ts_tablet_manager.cc:902] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:08.358765 2560 consensus_queue.cc:237] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:08.377574 4203 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:08.378230 2430 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:08.378310 7097 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:08.379565 7280 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:08.381832 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:16:08.382740 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57c4a133b5c646f78b07f37dc85904f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:08.384156 2299 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:08.384833 7097 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:08.448115 7097 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:08.452971 2299 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:08.453841 4203 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:08.462606 2152 catalog_manager.cc:5484] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:08.485296 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:08.509840 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:08.515226 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:16:08.538450 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 70431.4us
I20240503 14:16:08.539400 2430 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:08.555541 7449 tablet_copy_client.cc:308] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:08.556298 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 65a196c723884d068b3728e876c87eb2 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46595)
I20240503 14:16:08.556509 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 65a196c723884d068b3728e876c87eb2 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46595: session id = c2cf5d2244244a2db5ebc7dc46fd430d-65a196c723884d068b3728e876c87eb2
I20240503 14:16:08.559048 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:08.561246 2452 tablet_copy_source_session.cc:205] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:08.589023 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a196c723884d068b3728e876c87eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:08.614668 7449 tablet_copy_client.cc:783] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:08.614969 7449 tablet_copy_client.cc:647] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:08.617442 7449 tablet_copy_client.cc:523] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:08.626014 7449 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:08.734954 7449 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:08.738472 7449 tablet_bootstrap.cc:492] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:08.738787 7449 ts_tablet_manager.cc:1367] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.113s user 0.044s sys 0.012s
I20240503 14:16:08.739635 7449 raft_consensus.cc:348] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:08.739919 7449 raft_consensus.cc:729] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:08.744050 7449 consensus_queue.cc:260] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:08.765980 7449 ts_tablet_manager.cc:1398] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.027s user 0.002s sys 0.006s
I20240503 14:16:08.767411 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-65a196c723884d068b3728e876c87eb2 received from {username='slave'} at 127.20.99.8:46595
I20240503 14:16:08.767539 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-65a196c723884d068b3728e876c87eb2 on tablet 65a196c723884d068b3728e876c87eb2 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:08.803922 7515 ts_tablet_manager.cc:902] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:08.810747 4598 raft_consensus.cc:1203] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 14:16:08.845927 7515 tablet_copy_client.cc:308] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:16:08.846756 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet 57c4a133b5c646f78b07f37dc85904f6 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:41209)
I20240503 14:16:08.846933 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet 57c4a133b5c646f78b07f37dc85904f6 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:41209: session id = d4cbc83202c8473d8c897582d30e5989-57c4a133b5c646f78b07f37dc85904f6
I20240503 14:16:08.850997 2582 tablet_copy_source_session.cc:205] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:08.855397 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57c4a133b5c646f78b07f37dc85904f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:08.864293 7515 tablet_copy_client.cc:783] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:08.864583 7515 tablet_copy_client.cc:647] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:08.866621 7515 tablet_copy_client.cc:523] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:08.872473 7515 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:08.970304 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:09.000927 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.053611 7515 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:09.054068 7515 tablet_bootstrap.cc:492] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:09.055845 7515 ts_tablet_manager.cc:1367] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.182s user 0.061s sys 0.003s
I20240503 14:16:09.062580 7515 raft_consensus.cc:348] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:09.062987 7515 raft_consensus.cc:729] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:09.063354 7515 consensus_queue.cc:260] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:09.067504 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.069730 7515 ts_tablet_manager.cc:1398] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240503 14:16:09.117656 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.134202 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:16:09.169194 2497 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 93226.8us
I20240503 14:16:09.176324 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.202236 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
W20240503 14:16:09.207774 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 134207us
I20240503 14:16:09.208995 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-57c4a133b5c646f78b07f37dc85904f6 received from {username='slave'} at 127.20.99.7:41209
I20240503 14:16:09.209139 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-57c4a133b5c646f78b07f37dc85904f6 on tablet 57c4a133b5c646f78b07f37dc85904f6 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:09.213774 4469 raft_consensus.cc:1203] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:09.234385 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:09.600553 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.604194 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.616652 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:09.734046 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.737068 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:09.760462 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.766738 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:09.772325 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:09.780804 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.792107 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:09.802764 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.865747 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:09.943826 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:09.946277 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:09.968791 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:09.974548 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.100608 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.116379 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:10.124547 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:10.133050 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.146183 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:10.154012 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.160939 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.213665 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:10.221017 2691 consensus_queue.cc:237] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:10.245887 4598 raft_consensus.cc:1261] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:10.246531 7667 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:10.256565 2430 raft_consensus.cc:1261] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:10.257369 7667 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:10.273703 7568 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.277091 4598 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.290560 2159 catalog_manager.cc:5484] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.304863 2430 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.402843 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9974d80e8d31417788e10140de97e722 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:10.417344 7748 tablet_replica.cc:330] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:10.421386 7748 raft_consensus.cc:2229] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:10.421900 7748 raft_consensus.cc:2258] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:10.428511 7748 ts_tablet_manager.cc:1875] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:10.482575 4315 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:16:10.521855 2498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 77308.4us
I20240503 14:16:10.529349 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.537400 7748 ts_tablet_manager.cc:1888] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:10.541767 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:10.547155 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:10.548342 7748 log.cc:1196] T 9974d80e8d31417788e10140de97e722 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/9974d80e8d31417788e10140de97e722
I20240503 14:16:10.555893 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:10.577284 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:10.584718 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 9974d80e8d31417788e10140de97e722 (table rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]) successfully deleted
I20240503 14:16:10.591683 4579 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.655943 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.676549 2430 consensus_queue.cc:237] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:10.716867 4598 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:10.716922 4202 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:10.717581 7133 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:10.719034 4469 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:10.719761 7124 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:10.722918 7124 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:10.749658 2560 raft_consensus.cc:1261] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:10.750391 7133 consensus_queue.cc:1035] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:10.758555 7133 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.762898 4597 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.764317 4469 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.771116 2158 catalog_manager.cc:5484] T 65a196c723884d068b3728e876c87eb2 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.775851 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a196c723884d068b3728e876c87eb2 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:10.787549 2562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "059a4a35aa94412994bb431342ba0753"
dest_uuid: "b35c386497d046de9be5007a31ab7d10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38390
I20240503 14:16:10.787853 2562 raft_consensus.cc:595] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:10.797359 7808 tablet_replica.cc:330] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:10.805579 7808 raft_consensus.cc:2229] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:10.806193 7808 raft_consensus.cc:2258] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:10.808439 7808 ts_tablet_manager.cc:1875] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:10.821018 4202 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.831970 7808 ts_tablet_manager.cc:1888] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:10.836055 7808 log.cc:1196] T 65a196c723884d068b3728e876c87eb2 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/65a196c723884d068b3728e876c87eb2
I20240503 14:16:10.836514 2562 raft_consensus.cc:2932] T 65a196c723884d068b3728e876c87eb2 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.840601 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet 65a196c723884d068b3728e876c87eb2 (table rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]) successfully deleted
I20240503 14:16:10.898975 7762 raft_consensus.cc:979] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: : Instructing follower c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:10.899142 7762 raft_consensus.cc:1067] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Signalling peer c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:10.900453 4597 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "059a4a35aa94412994bb431342ba0753"
dest_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
from {username='slave'} at 127.20.99.3:59567
I20240503 14:16:10.900774 4597 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.900949 4597 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:10.907835 4597 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:10.910305 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:10.910504 2691 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:10.913350 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:10.915128 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:10.915416 2301 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:10.915743 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:10.915951 2562 raft_consensus.cc:3032] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:10.916051 2562 raft_consensus.cc:729] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:16:10.916446 2562 consensus_queue.cc:260] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:10.916976 2562 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:10.918200 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:10.919337 4533 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c2cf5d2244244a2db5ebc7dc46fd430d, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:10.919749 7726 raft_consensus.cc:2781] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:10.923240 7726 raft_consensus.cc:686] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Becoming Leader. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Running, Role: LEADER
I20240503 14:16:10.923812 7726 consensus_queue.cc:237] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:10.929153 2158 catalog_manager.cc:5484] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: term changed from 1 to 2, leader changed from b35c386497d046de9be5007a31ab7d10 (127.20.99.3) to c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8). New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:10.940708 4597 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:16:10.945029 2562 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:10.946290 4202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7146de39e348442da4a363e95fbe0789"
I20240503 14:16:10.946671 4202 raft_consensus.cc:3037] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:10.950002 4202 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:10.982757 2430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304"
dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49846
I20240503 14:16:10.983052 2430 raft_consensus.cc:595] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:11.022818 7124 raft_consensus.cc:979] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: : Instructing follower 285488446ac445c5a5dfe654bfaf9835 to start an election
I20240503 14:16:11.023054 7124 raft_consensus.cc:1067] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Signalling peer 285488446ac445c5a5dfe654bfaf9835 to start an election
I20240503 14:16:11.024451 4333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304"
dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
from {username='slave'} at 127.20.99.2:37851
I20240503 14:16:11.024837 4333 raft_consensus.cc:482] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.025074 4333 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:11.030951 4333 raft_consensus.cc:504] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:11.033149 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:16:11.033159 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:11.033181 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:11.033361 2301 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:11.033371 2430 raft_consensus.cc:3032] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:11.033397 2562 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:11.033474 2430 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:16:11.033792 2430 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:11.034349 2430 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:11.036298 2562 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:11.036804 2301 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:11.037278 4269 leader_election.cc:302] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:16:11.037659 2430 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:11.047667 4333 leader_election.cc:288] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:16:11.048928 7861 raft_consensus.cc:2781] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:11.098860 4469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b359fb8d9afc4c9c856897a722100304" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
I20240503 14:16:11.099305 4469 raft_consensus.cc:3037] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:11.102605 4469 raft_consensus.cc:2454] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:11.104631 7861 raft_consensus.cc:686] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Becoming Leader. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Running, Role: LEADER
I20240503 14:16:11.105654 7861 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:11.126545 2158 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: term changed from 1 to 2, leader changed from 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) to 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6). New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:11.283243 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:11.298754 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:11.322305 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.348060 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.362509 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:11.393646 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:11.405617 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.428560 2691 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.429796 7726 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.435149 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:11.454540 2300 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.455746 7726 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.498172 2562 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.500103 7726 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.502137 2562 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.503829 7861 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.509116 4202 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.509846 7726 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.579473 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:16:11.613415 4533 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 56694.5us
I20240503 14:16:11.616631 4469 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.617250 7861 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.618659 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:11.649258 2431 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.649926 7861 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:11.665802 4534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 69041.5us
I20240503 14:16:11.674492 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.681424 2301 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:11.682169 7993 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.685289 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:11.697607 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:11.727986 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:11.749639 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:11.801198 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.884843 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:11.919549 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.927842 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:11.943924 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:11.977650 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:11.998656 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:12.024024 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:12.032303 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:12.036993 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:12.088774 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:12.098323 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:12.104115 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:12.112651 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:12.135149 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:12.147976 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:12.155227 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:12.181713 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:12.199064 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:12.213003 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:12.234810 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:12.235229 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:12.239847 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:12.268338 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:12.340387 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:12.789691 2301 consensus_queue.cc:237] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:12.793120 2431 raft_consensus.cc:1261] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:12.793473 4203 raft_consensus.cc:1261] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:12.794298 7696 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:12.796974 7696 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:12.815809 7696 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:12.818475 2431 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:12.825811 2159 catalog_manager.cc:5484] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:12.826876 4202 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:12.881022 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 944d2a983d6247498b02a6e7b7daf746 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:12.938203 8139 tablet_replica.cc:330] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:12.939343 8139 raft_consensus.cc:2229] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:12.940762 8139 raft_consensus.cc:2258] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:12.941900 8139 ts_tablet_manager.cc:1875] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:12.958812 8139 ts_tablet_manager.cc:1888] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:12.958976 8139 log.cc:1196] T 944d2a983d6247498b02a6e7b7daf746 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/944d2a983d6247498b02a6e7b7daf746
I20240503 14:16:12.960001 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 944d2a983d6247498b02a6e7b7daf746 (table rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]) successfully deleted
I20240503 14:16:12.965354 2562 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:12.969691 2690 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:12.970355 8044 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:12.971633 4333 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:12.972272 8049 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:12.973752 8049 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:12.975677 4335 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:12.978144 2690 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:12.989506 2153 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:12.993818 2562 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:12.996304 4469 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:12.997018 7994 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:12.999277 7994 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:13.000898 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 32927a23842b4725b9dfafbac47fe754 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:13.038007 2299 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:13.038878 7994 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:13.054625 8172 tablet_replica.cc:330] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
W20240503 14:16:13.058138 8162 log.cc:924] Time spent T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240503 14:16:13.058985 7994 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:13.063544 4597 consensus_queue.cc:237] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:13.068234 8172 raft_consensus.cc:2229] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:13.072332 8172 raft_consensus.cc:2258] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:13.071955 4335 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:13.074385 4202 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:13.075227 2690 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:13.075837 7968 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:13.078658 7968 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:13.078697 4469 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:13.079420 2301 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:13.084893 8172 ts_tablet_manager.cc:1875] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:13.086753 2299 raft_consensus.cc:1261] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:13.088227 2155 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 5 to 7, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:13.094966 7968 consensus_queue.cc:1035] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:13.103260 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet b359fb8d9afc4c9c856897a722100304 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:13.111340 8172 ts_tablet_manager.cc:1888] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:13.111578 8172 log.cc:1196] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/32927a23842b4725b9dfafbac47fe754
I20240503 14:16:13.118433 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 32927a23842b4725b9dfafbac47fe754 (table rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]) successfully deleted
I20240503 14:16:13.118664 8172 tablet_replica.cc:330] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:13.119647 8172 raft_consensus.cc:2229] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:13.119932 8172 raft_consensus.cc:2258] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:13.121649 8172 ts_tablet_manager.cc:1875] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:13.121809 2562 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:13.136379 8172 ts_tablet_manager.cc:1888] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:13.136565 8172 log.cc:1196] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/b359fb8d9afc4c9c856897a722100304
I20240503 14:16:13.137440 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet b359fb8d9afc4c9c856897a722100304 (table rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]) successfully deleted
I20240503 14:16:13.139969 8206 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:13.141970 4202 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:13.143975 2299 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:13.152503 2153 catalog_manager.cc:5484] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:13.157408 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 059a4a35aa94412994bb431342ba0753 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:13.157868 8139 tablet_replica.cc:330] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:13.158072 8139 raft_consensus.cc:2229] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:13.158324 8139 raft_consensus.cc:2258] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:13.159070 8139 ts_tablet_manager.cc:1875] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:13.160621 2690 raft_consensus.cc:2932] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:13.166599 8139 ts_tablet_manager.cc:1888] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:13.176066 8139 log.cc:1196] T 059a4a35aa94412994bb431342ba0753 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/059a4a35aa94412994bb431342ba0753
I20240503 14:16:13.178525 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 059a4a35aa94412994bb431342ba0753 (table rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]) successfully deleted
I20240503 14:16:13.251374 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:13.273335 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:13.396706 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:13.429258 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:13.438819 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:13.496922 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:13.540496 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:13.796444 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:16:13.895697 2497 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 54587.7us
I20240503 14:16:14.156721 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.211869 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:14.212306 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.228812 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:14.264406 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.285651 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.293207 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.300982 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.312238 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:14.346398 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.403400 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:14.405520 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:14.416468 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.416821 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.417806 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:14.435307 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.443420 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:14.474138 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.515758 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.565635 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.568431 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.581770 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:14.592641 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:14.653656 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.671710 2560 consensus_queue.cc:237] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:14.689833 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.698216 4469 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:14.699193 8295 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:14.699800 4202 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:14.701117 8295 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:14.705153 2300 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:14.711863 8403 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:14.721153 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:14.722322 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.746742 8403 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:14.766407 2300 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:14.766981 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:14.786693 2155 catalog_manager.cc:5484] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:14.809573 4202 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:14.810180 4467 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:14.811712 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:14.837284 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:14.841683 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:14.865326 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:14.925187 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 57c4a133b5c646f78b07f37dc85904f6 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:14.954527 8428 tablet_replica.cc:330] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:14.956463 8428 raft_consensus.cc:2229] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:14.967739 8428 raft_consensus.cc:2258] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:14.978519 8428 ts_tablet_manager.cc:1875] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.014391 8428 ts_tablet_manager.cc:1888] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:15.014693 8428 log.cc:1196] T 57c4a133b5c646f78b07f37dc85904f6 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/57c4a133b5c646f78b07f37dc85904f6
I20240503 14:16:15.015559 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 57c4a133b5c646f78b07f37dc85904f6 (table rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]) successfully deleted
I20240503 14:16:15.126893 2690 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:15.138906 4467 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:15.139001 2562 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:15.139743 8430 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:15.140193 7568 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:15.144403 2560 consensus_queue.cc:237] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:15.155859 4597 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:15.156625 8295 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:15.156322 8430 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:15.160091 4469 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:15.163827 4202 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:15.164589 8295 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:15.172933 2158 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:15.175149 2690 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:15.176280 8295 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:15.178897 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet f139d77b0f0c47809e6408b2771adb73 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:15.201793 2560 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:15.209275 8493 tablet_replica.cc:330] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:15.209635 8493 raft_consensus.cc:2229] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.210188 8493 raft_consensus.cc:2258] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.211156 8493 ts_tablet_manager.cc:1875] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.216181 8411 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:15.217115 4202 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:15.224236 2690 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:15.226272 4598 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:15.228193 2153 catalog_manager.cc:5484] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:15.239353 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 6325a400641b4d25b95592abb4af7d2d with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:15.239732 8428 tablet_replica.cc:330] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:15.239928 8428 raft_consensus.cc:2229] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.240290 8428 raft_consensus.cc:2258] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.241788 8428 ts_tablet_manager.cc:1875] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.247129 8428 ts_tablet_manager.cc:1888] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:15.247324 8428 log.cc:1196] T 6325a400641b4d25b95592abb4af7d2d P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/6325a400641b4d25b95592abb4af7d2d
I20240503 14:16:15.248476 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 6325a400641b4d25b95592abb4af7d2d (table rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]) successfully deleted
I20240503 14:16:15.265326 8493 ts_tablet_manager.cc:1888] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:15.273386 8493 log.cc:1196] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/f139d77b0f0c47809e6408b2771adb73
I20240503 14:16:15.274670 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet f139d77b0f0c47809e6408b2771adb73 (table rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]) successfully deleted
I20240503 14:16:15.468500 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:15.479539 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:15.498931 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:15.526227 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:15.546057 2690 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:15.560544 4202 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:15.561404 7568 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:15.567101 2431 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:15.567989 8430 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:15.570195 8430 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:15.574333 4203 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:15.576666 2153 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:15.583041 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b63f87fb384bb2a48044bcad87c282 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:15.598527 8579 tablet_replica.cc:330] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:15.598768 8579 raft_consensus.cc:2229] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:15.599025 8579 raft_consensus.cc:2258] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:15.600138 8579 ts_tablet_manager.cc:1875] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:15.603428 2671 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:16:15.608546 8579 ts_tablet_manager.cc:1888] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:15.608692 8579 log.cc:1196] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/61b63f87fb384bb2a48044bcad87c282
I20240503 14:16:15.609649 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 61b63f87fb384bb2a48044bcad87c282 (table rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]) successfully deleted
I20240503 14:16:15.629287 4449 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:15.656745 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:15.707648 2431 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:15.719781 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.011221 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.011386 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:16.053546 2691 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42278
I20240503 14:16:16.053891 2691 raft_consensus.cc:595] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:16.064010 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:16.065033 7568 raft_consensus.cc:979] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: : Instructing follower d4cbc83202c8473d8c897582d30e5989 to start an election
I20240503 14:16:16.065217 7568 raft_consensus.cc:1067] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Signalling peer d4cbc83202c8473d8c897582d30e5989 to start an election
I20240503 14:16:16.068682 4469 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe"
dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
from {username='slave'} at 127.20.99.4:40293
I20240503 14:16:16.069018 4469 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:16.069154 4469 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:16.071341 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:16.073182 4469 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:16.076656 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:16:16.076922 2431 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:16.081979 2431 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:16.085980 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:16.085986 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:16.086253 2691 raft_consensus.cc:3032] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:16.086225 2300 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:16.086395 2691 raft_consensus.cc:729] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:16.086800 2691 consensus_queue.cc:260] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:16.087568 2691 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:16.090009 2300 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:16.091661 2691 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:16.100541 4402 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829, d4cbc83202c8473d8c897582d30e5989; no voters:
I20240503 14:16:16.109642 4469 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 2 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:16.154932 8647 raft_consensus.cc:2781] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:16:16.163044 4402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 62559.6us
I20240503 14:16:16.164398 8647 raft_consensus.cc:686] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 2 LEADER]: Becoming Leader. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Running, Role: LEADER
I20240503 14:16:16.165292 8647 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:16.226282 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:16.268237 4597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
I20240503 14:16:16.268589 4597 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:16.271879 4597 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:16.277303 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.297344 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.297243 2159 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 reported cstate change: term changed from 1 to 2, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to d4cbc83202c8473d8c897582d30e5989 (127.20.99.7). New cstate: current_term: 2 leader_uuid: "d4cbc83202c8473d8c897582d30e5989" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:16.327404 4203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78edc267e9f44d82b7dbb663628359fe" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7146de39e348442da4a363e95fbe0789"
I20240503 14:16:16.327675 4203 raft_consensus.cc:3037] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:16.331882 4203 raft_consensus.cc:2454] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:16.356158 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:16.449692 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:16.466396 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:16.473268 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:16.483192 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.491654 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.497900 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:16.505599 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.534329 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:16.551800 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:16.556733 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:16.577136 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.578135 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:16.610818 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:16.667235 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.681793 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:16.682214 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:16.683647 2690 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:16.684731 8647 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:16.801012 2431 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:16.801749 8647 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:16.863492 4203 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:16.865434 8647 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:16.867509 2300 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:16.868341 8647 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:16.941515 4597 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:16.949348 8757 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:16.964000 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:17.007782 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:16:17.040032 2368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 61601.8us
W20240503 14:16:17.049727 4405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 107512us
I20240503 14:16:17.070730 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:17.095600 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:17.107640 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:17.136754 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:17.163165 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:17.192229 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:17.212985 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:17.213677 2412 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:17.250247 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:17.327998 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:17.328825 2280 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:17.333813 4579 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:17.340934 2300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc"
dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59416
I20240503 14:16:17.341536 2300 raft_consensus.cc:595] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:17.354313 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:17.397661 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:17.408757 2430 consensus_queue.cc:237] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:17.421725 4202 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:17.422369 8780 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:17.429370 4597 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:17.430383 2691 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:17.430912 4335 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:17.431025 8780 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:17.431915 8779 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:17.432559 8779 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:17.443490 2300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740"
dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59426
I20240503 14:16:17.443728 2300 raft_consensus.cc:595] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:17.448218 7696 raft_consensus.cc:979] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: : Instructing follower c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:17.448441 7696 raft_consensus.cc:1067] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Signalling peer c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:17.451668 4598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc"
dest_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
from {username='slave'} at 127.20.99.1:45011
I20240503 14:16:17.451917 4598 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:17.452040 4598 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.455560 4598 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:17.479374 8779 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:17.485013 4597 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:17.486377 2159 catalog_manager.cc:5484] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 6 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:17.496405 4202 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:17.512336 2690 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:17.520596 4598 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:17.521497 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:17.521723 2300 raft_consensus.cc:3032] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:17.521802 2300 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:16:17.522178 2300 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:17.522603 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:16:17.522781 2430 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.522794 2300 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.522485 4335 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:17.526124 2430 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.533020 2300 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.533903 4533 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f, c2cf5d2244244a2db5ebc7dc46fd430d; no voters:
I20240503 14:16:17.534341 8205 raft_consensus.cc:2781] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:17.542271 8205 raft_consensus.cc:686] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Becoming Leader. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Running, Role: LEADER
I20240503 14:16:17.542729 8205 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:17.614699 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:17.614967 2690 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.630007 2690 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.709029 7696 raft_consensus.cc:979] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: : Instructing follower c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:17.709211 7696 raft_consensus.cc:1067] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Signalling peer c2cf5d2244244a2db5ebc7dc46fd430d to start an election
W20240503 14:16:17.710230 2625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:44387, user_credentials={real_user=slave}} blocked reactor thread for 116895us
W20240503 14:16:17.713258 2626 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 116365us
I20240503 14:16:17.713199 4598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740"
dest_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
from {username='slave'} at 127.20.99.1:45011
I20240503 14:16:17.713474 4598 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:17.713618 4598 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.718619 4598 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:17.722239 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f631c7713d24f4caf7550bb05147615 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:17.727584 2155 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: term changed from 1 to 2, leader changed from 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) to c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8). New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:17.738339 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:17.738615 2560 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.745697 2560 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.747874 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:17.748237 2300 raft_consensus.cc:3032] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:17.748349 2300 raft_consensus.cc:729] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:16:17.748728 2300 consensus_queue.cc:260] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:17.749264 2300 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.752789 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
I20240503 14:16:17.753209 4335 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:16:17.753821 2628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 162776us
I20240503 14:16:17.755311 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d584b7026b64ba1b720ed21d7019740" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:17.755613 2690 raft_consensus.cc:3037] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.759217 2690 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.760318 4598 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:16:17.759821 4335 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.761003 4532 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c2cf5d2244244a2db5ebc7dc46fd430d, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:17.767431 8205 raft_consensus.cc:2781] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:17.767611 8205 raft_consensus.cc:686] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Becoming Leader. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Running, Role: LEADER
I20240503 14:16:17.767915 8205 consensus_queue.cc:237] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:17.777282 2300 raft_consensus.cc:2454] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:17.790283 8857 tablet_replica.cc:330] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:17.790503 8857 raft_consensus.cc:2229] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:17.790800 8857 raft_consensus.cc:2258] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:17.796471 8857 ts_tablet_manager.cc:1875] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:17.799924 2155 catalog_manager.cc:5484] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: term changed from 1 to 2, leader changed from 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) to c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8). New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:17.813418 8857 ts_tablet_manager.cc:1888] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:17.813624 8857 log.cc:1196] T 5f631c7713d24f4caf7550bb05147615 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f631c7713d24f4caf7550bb05147615
I20240503 14:16:17.814502 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 5f631c7713d24f4caf7550bb05147615 (table rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]) successfully deleted
I20240503 14:16:17.852244 2690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59720
I20240503 14:16:17.852502 2690 raft_consensus.cc:595] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:17.872841 8430 raft_consensus.cc:979] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: : Instructing follower 685c6181561b4056abdecfd6d8bdb3b2 to start an election
I20240503 14:16:17.873019 8430 raft_consensus.cc:1067] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Signalling peer 685c6181561b4056abdecfd6d8bdb3b2 to start an election
I20240503 14:16:17.873991 2431 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd"
dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
from {username='slave'} at 127.20.99.4:45821
I20240503 14:16:17.874284 2431 raft_consensus.cc:482] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:17.874395 2431 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.878019 2431 raft_consensus.cc:504] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:17.879860 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:17.880169 2300 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.884012 2300 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:17.885727 2431 leader_election.cc:288] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:17.886163 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:17.886446 2690 raft_consensus.cc:3032] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:17.886538 2690 raft_consensus.cc:729] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:17.886857 2690 consensus_queue.cc:260] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:17.887439 2690 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.890991 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b112f8fb89c7404b87a4c6e3d72fdcfd" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
I20240503 14:16:17.891278 4335 raft_consensus.cc:3037] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:17.895011 4335 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:17.917665 2368 leader_election.cc:302] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f; no voters:
I20240503 14:16:17.918040 8779 raft_consensus.cc:2781] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:17.918181 8779 raft_consensus.cc:686] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:16:17.918545 8779 consensus_queue.cc:237] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:17.923341 2690 raft_consensus.cc:2454] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:17.966363 2155 catalog_manager.cc:5484] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 1 to 2, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:17.975149 2300 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:17.976904 8205 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:18.052165 4335 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:18.053109 8205 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:18.055732 2690 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:18.058261 8842 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:18.127444 2431 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:18.136435 8842 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:18.151871 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:18.160977 2562 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:18.161691 8205 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:18.162079 2300 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:18.162536 8205 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:18.263650 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:18.264245 2690 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:18.265106 8842 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:18.281661 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:18.314939 2300 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:18.315735 8779 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:18.356122 2280 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:18.390481 4334 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:18.391275 8779 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:18.405795 2690 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:18.406842 8779 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:18.409296 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:18.426738 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:18.445365 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:18.521162 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:18.523190 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:18.523728 2671 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:16:18.530382 4467 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:18.554169 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:18.558426 2280 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:18.581267 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:18.587324 2300 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:18.588064 8764 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:18.595741 4598 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:18.596364 8764 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:18.607901 4202 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:18.608625 8764 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:18.648742 2431 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:18.658588 8764 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:18.659015 4404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 50584us
W20240503 14:16:18.713371 4403 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 54050.3us
I20240503 14:16:18.718564 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:18.724503 8764 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:18.731110 2155 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "d4cbc83202c8473d8c897582d30e5989" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:18.732115 4597 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:18.745339 4202 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:16:18.776656 4403 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 57073.5us
I20240503 14:16:18.780037 2300 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:18.791782 2431 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:18.800472 2671 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:16:18.811933 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:18.865849 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet 78edc267e9f44d82b7dbb663628359fe with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:18.908473 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:18.927361 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:18.943878 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:18.944397 9040 tablet_replica.cc:330] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:18.944945 9040 raft_consensus.cc:2229] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:18.945226 9040 raft_consensus.cc:2258] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:18.946730 9040 ts_tablet_manager.cc:1875] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:18.969619 2280 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:18.991849 9040 ts_tablet_manager.cc:1888] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:16:18.992204 9040 log.cc:1196] T 78edc267e9f44d82b7dbb663628359fe P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/78edc267e9f44d82b7dbb663628359fe
I20240503 14:16:18.993242 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet 78edc267e9f44d82b7dbb663628359fe (table rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]) successfully deleted
I20240503 14:16:19.009642 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:19.013852 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:19.050828 2671 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:16:19.085075 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:19.166290 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:19.175139 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:19.203754 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:19.257774 2280 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:19.284972 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:19.297523 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:19.297717 2671 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:16:19.335251 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:19.473731 4598 consensus_queue.cc:237] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:19.518811 2561 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:19.519135 2690 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:19.520016 8205 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:19.520751 8205 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:19.571702 9129 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:19.572562 2690 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:19.578676 2159 catalog_manager.cc:5484] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:16:19.626117 4532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 48899us
I20240503 14:16:19.629431 2561 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:19.672510 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d584b7026b64ba1b720ed21d7019740 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:19.684526 9161 tablet_replica.cc:330] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:19.684823 9161 raft_consensus.cc:2229] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:19.685055 9161 raft_consensus.cc:2258] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:19.691512 2300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36"
dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59492
I20240503 14:16:19.691782 2300 raft_consensus.cc:595] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:19.694439 9130 raft_consensus.cc:979] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: : Instructing follower c37183c2965f46a28847c1262eb97829 to start an election
I20240503 14:16:19.694581 9130 raft_consensus.cc:1067] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Signalling peer c37183c2965f46a28847c1262eb97829 to start an election
I20240503 14:16:19.695549 2690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
from {username='slave'} at 127.20.99.1:48499
I20240503 14:16:19.695853 2690 raft_consensus.cc:482] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:19.695986 2690 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:19.699831 4598 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:19.700758 2690 raft_consensus.cc:504] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:19.704155 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:19.704463 2300 raft_consensus.cc:3032] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:19.704553 2300 raft_consensus.cc:729] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:16:19.704874 2300 consensus_queue.cc:260] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:19.705415 2300 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:19.708698 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:16:19.708957 2431 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:19.712299 2431 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
I20240503 14:16:19.718426 4334 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:19.719532 9146 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:19.721954 2300 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
I20240503 14:16:19.722659 2626 leader_election.cc:302] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:19.723062 9081 raft_consensus.cc:2781] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:19.723209 9081 raft_consensus.cc:686] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 LEADER]: Becoming Leader. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:19.723582 9081 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:19.726505 2690 leader_election.cc:288] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:16:19.730924 4467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
I20240503 14:16:19.731220 4467 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:19.737411 4467 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
I20240503 14:16:19.743227 2431 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:19.744160 8205 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:19.757364 2690 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:19.759261 9146 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:19.763013 8205 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:19.773891 4334 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:19.784485 2691 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:19.784703 2157 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 5 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:19.788846 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0eebf47c494dad8edf06197cdf15bc with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:19.807484 9161 ts_tablet_manager.cc:1875] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:19.823313 2151 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 reported cstate change: term changed from 1 to 2, leader changed from 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) to c37183c2965f46a28847c1262eb97829 (127.20.99.4). New cstate: current_term: 2 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:19.825729 2430 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:19.851928 9161 ts_tablet_manager.cc:1888] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:16:19.853643 9161 log.cc:1196] T 1d584b7026b64ba1b720ed21d7019740 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d584b7026b64ba1b720ed21d7019740
I20240503 14:16:19.861335 9161 tablet_replica.cc:330] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:19.861377 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet 1d584b7026b64ba1b720ed21d7019740 (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
I20240503 14:16:19.861661 9161 raft_consensus.cc:2229] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:19.861900 9161 raft_consensus.cc:2258] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:19.862691 9161 ts_tablet_manager.cc:1875] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:19.875485 9161 ts_tablet_manager.cc:1888] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:19.875700 9161 log.cc:1196] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe0eebf47c494dad8edf06197cdf15bc
I20240503 14:16:19.881183 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet fe0eebf47c494dad8edf06197cdf15bc (table rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]) successfully deleted
I20240503 14:16:19.969506 2430 consensus_queue.cc:237] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:19.973119 2300 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:19.974498 8912 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:19.992477 8912 raft_consensus.cc:2932] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:19.996649 2299 raft_consensus.cc:2932] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:20.004976 2151 catalog_manager.cc:5484] T b112f8fb89c7404b87a4c6e3d72fdcfd P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 6, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:20.005473 4334 raft_consensus.cc:1261] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:20.006008 8912 consensus_queue.cc:1035] T b112f8fb89c7404b87a4c6e3d72fdcfd P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:20.014263 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet b112f8fb89c7404b87a4c6e3d72fdcfd with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:20.023998 4334 raft_consensus.cc:2932] T b112f8fb89c7404b87a4c6e3d72fdcfd P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:20.039054 9225 tablet_replica.cc:330] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:20.039446 9225 raft_consensus.cc:2229] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:20.039750 9225 raft_consensus.cc:2258] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:20.040827 9225 ts_tablet_manager.cc:1875] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:20.073371 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:20.107337 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:20.107439 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:20.117462 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:20.133132 9225 ts_tablet_manager.cc:1888] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:16:20.147686 2430 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:20.148859 8854 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:20.156141 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:20.172257 9225 log.cc:1196] T b112f8fb89c7404b87a4c6e3d72fdcfd P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/b112f8fb89c7404b87a4c6e3d72fdcfd
I20240503 14:16:20.180408 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:20.181097 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:20.221871 2299 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:20.222486 8854 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:20.229354 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet b112f8fb89c7404b87a4c6e3d72fdcfd (table rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]) successfully deleted
I20240503 14:16:20.235102 4467 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:20.235668 9081 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:20.240063 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:20.384048 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:20.387544 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:20.416803 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:20.460700 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:20.475694 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:20.527736 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:20.532065 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:20.545833 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:20.819720 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:20.828855 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:20.852563 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:20.864835 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:20.887486 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:20.888499 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:20.907290 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:20.967299 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:16:21.057607 2235 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 50199.3us
I20240503 14:16:21.188364 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:21.204582 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:21.238962 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:21.243937 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:21.288583 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:21.325573 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:21.336575 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:21.360236 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:21.364071 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:21.393999 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:21.396566 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:21.406085 2690 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:21.418891 4469 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:21.419159 2430 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:21.420007 8854 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:21.420326 9081 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:21.445400 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:21.459681 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:21.481675 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:21.555792 8854 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:21.561420 4467 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:21.561853 2430 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:21.567123 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:21.572022 2157 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:21.583388 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:21.702431 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 75bafc7675ca45639b696db6e4b7ce36 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:21.709009 9463 tablet_replica.cc:330] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:21.709223 9463 raft_consensus.cc:2229] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:21.709470 9463 raft_consensus.cc:2258] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:21.714871 9463 ts_tablet_manager.cc:1875] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:21.745872 9463 ts_tablet_manager.cc:1888] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:16:21.746935 9463 log.cc:1196] T 75bafc7675ca45639b696db6e4b7ce36 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/75bafc7675ca45639b696db6e4b7ce36
I20240503 14:16:21.747706 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet 75bafc7675ca45639b696db6e4b7ce36 (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
I20240503 14:16:21.775861 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:21.820724 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:21.821135 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:21.852599 4468 consensus_queue.cc:237] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:21.855705 4202 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:16:21.855846 4598 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:16:21.856355 9503 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:21.856750 9013 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:21.864454 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:21.872764 2300 raft_consensus.cc:1261] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:16:21.877341 9013 consensus_queue.cc:1035] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:21.928514 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:21.934008 2368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 94443.4us
I20240503 14:16:21.955044 9504 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:21.955791 4598 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:21.961764 2299 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:21.964490 2153 catalog_manager.cc:5484] T 78edc267e9f44d82b7dbb663628359fe P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 8 to 9, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "d4cbc83202c8473d8c897582d30e5989" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:21.975580 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:21.983405 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 78edc267e9f44d82b7dbb663628359fe with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39122
W20240503 14:16:21.993614 2368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 56524us
I20240503 14:16:22.003274 2410 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:22.005102 4202 raft_consensus.cc:2932] T 78edc267e9f44d82b7dbb663628359fe P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.029804 9516 tablet_replica.cc:330] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:22.030185 9516 raft_consensus.cc:2229] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:22.030479 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:22.031704 9516 raft_consensus.cc:2258] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:22.032529 9516 ts_tablet_manager.cc:1875] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:22.062455 9516 ts_tablet_manager.cc:1888] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 14:16:22.062709 9516 log.cc:1196] T 78edc267e9f44d82b7dbb663628359fe P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/78edc267e9f44d82b7dbb663628359fe
I20240503 14:16:22.063558 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 78edc267e9f44d82b7dbb663628359fe (table rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]) successfully deleted
I20240503 14:16:22.138958 4598 consensus_queue.cc:237] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:22.142328 2561 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:22.142951 9146 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:22.153203 2299 consensus_queue.cc:237] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:22.161728 4202 raft_consensus.cc:1261] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:22.162326 9436 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:22.163619 9550 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.165181 2561 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.172636 2156 catalog_manager.cc:5484] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.186336 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d584b7026b64ba1b720ed21d7019740 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:22.214375 9560 tablet_replica.cc:330] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:22.214823 9560 raft_consensus.cc:2229] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:22.215634 9560 raft_consensus.cc:2258] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:22.221841 4202 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:22.228893 2156 catalog_manager.cc:5484] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 reported cstate change: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:22.237083 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 944d2a983d6247498b02a6e7b7daf746 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:22.237589 9516 tablet_replica.cc:330] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:22.237821 9516 raft_consensus.cc:2229] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:22.238179 9516 raft_consensus.cc:2258] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:22.239113 9516 ts_tablet_manager.cc:1875] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:22.243484 4597 consensus_queue.cc:237] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:22.220855 9558 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:22.245083 9560 ts_tablet_manager.cc:1875] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:22.247519 2561 raft_consensus.cc:1261] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 14:16:22.248523 4534 consensus_peers.cc:485] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d584b7026b64ba1b720ed21d7019740. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:22.249397 9146 consensus_queue.cc:1035] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:22.253091 9146 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:22.256039 2561 raft_consensus.cc:2932] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:22.260632 9560 ts_tablet_manager.cc:1888] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:22.260768 9560 log.cc:1196] T 1d584b7026b64ba1b720ed21d7019740 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/1d584b7026b64ba1b720ed21d7019740
I20240503 14:16:22.263190 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1d584b7026b64ba1b720ed21d7019740 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:22.265388 2156 catalog_manager.cc:5484] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:22.268379 9516 ts_tablet_manager.cc:1888] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:22.268507 9516 log.cc:1196] T 944d2a983d6247498b02a6e7b7daf746 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/944d2a983d6247498b02a6e7b7daf746
I20240503 14:16:22.271934 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 944d2a983d6247498b02a6e7b7daf746 (table rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]) successfully deleted
I20240503 14:16:22.273283 2299 consensus_queue.cc:237] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:16:22.278581 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 944d2a983d6247498b02a6e7b7daf746. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:22.280083 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet 1d584b7026b64ba1b720ed21d7019740 (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
I20240503 14:16:22.282812 4202 raft_consensus.cc:1261] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:22.284543 9558 consensus_queue.cc:1035] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:22.291949 9558 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.293641 4202 raft_consensus.cc:2932] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.297839 2145 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 944d2a983d6247498b02a6e7b7daf746 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:22.300282 2151 catalog_manager.cc:5484] T 944d2a983d6247498b02a6e7b7daf746 P 7146de39e348442da4a363e95fbe0789 reported cstate change: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:22.621654 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:22.649324 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:22.700263 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:22.704885 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:22.707453 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:22.687039 9651 ts_tablet_manager.cc:902] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
I20240503 14:16:22.739974 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:22.745908 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:22.757287 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:22.759409 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:22.815560 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:22.824501 9667 ts_tablet_manager.cc:902] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:22.828078 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:22.853183 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:22.888649 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:22.900707 2690 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:16:22.921486 2561 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:22.929296 8854 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:22.946604 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:22.963394 2671 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:22.963829 2670 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:22.968492 9651 tablet_copy_client.cc:308] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.1:33597
I20240503 14:16:22.970263 2320 tablet_copy_service.cc:140] P 7f22b4d5b1b5466dbbdd344f0721c05f: Received BeginTabletCopySession request for tablet 944d2a983d6247498b02a6e7b7daf746 from peer c2cf5d2244244a2db5ebc7dc46fd430d ({username='slave'} at 127.20.99.8:46371)
I20240503 14:16:22.970444 2320 tablet_copy_service.cc:161] P 7f22b4d5b1b5466dbbdd344f0721c05f: Beginning new tablet copy session on tablet 944d2a983d6247498b02a6e7b7daf746 from peer c2cf5d2244244a2db5ebc7dc46fd430d at {username='slave'} at 127.20.99.8:46371: session id = c2cf5d2244244a2db5ebc7dc46fd430d-944d2a983d6247498b02a6e7b7daf746
I20240503 14:16:22.973579 2320 tablet_copy_source_session.cc:205] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:22.975294 9651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944d2a983d6247498b02a6e7b7daf746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:22.983877 8854 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:22.991832 2561 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:22.992455 2156 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 5 to 6, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:23.006371 9667 tablet_copy_client.cc:308] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.8:45929
I20240503 14:16:23.084203 4619 tablet_copy_service.cc:140] P c2cf5d2244244a2db5ebc7dc46fd430d: Received BeginTabletCopySession request for tablet 1d584b7026b64ba1b720ed21d7019740 from peer 7146de39e348442da4a363e95fbe0789 ({username='slave'} at 127.20.99.5:52931)
I20240503 14:16:23.084527 4619 tablet_copy_service.cc:161] P c2cf5d2244244a2db5ebc7dc46fd430d: Beginning new tablet copy session on tablet 1d584b7026b64ba1b720ed21d7019740 from peer 7146de39e348442da4a363e95fbe0789 at {username='slave'} at 127.20.99.5:52931: session id = 7146de39e348442da4a363e95fbe0789-1d584b7026b64ba1b720ed21d7019740
I20240503 14:16:23.094651 4619 tablet_copy_source_session.cc:205] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:23.096849 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d584b7026b64ba1b720ed21d7019740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:23.107443 9651 tablet_copy_client.cc:783] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:23.107793 9651 tablet_copy_client.cc:647] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:23.107800 9667 tablet_copy_client.cc:783] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:23.108101 9667 tablet_copy_client.cc:647] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:23.115249 9651 tablet_copy_client.cc:523] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:23.117564 9667 tablet_copy_client.cc:523] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:23.120890 9651 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap starting.
I20240503 14:16:23.127869 9667 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: Bootstrap starting.
I20240503 14:16:23.165767 2690 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:23.169361 2561 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:23.170332 4468 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:23.171448 9081 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:23.172963 9081 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:23.178083 4468 raft_consensus.cc:858] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
W20240503 14:16:23.178256 4468 raft_consensus.cc:1565] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:23.179427 4449 tablet_service.cc:1514] Processing DeleteTablet for tablet f139d77b0f0c47809e6408b2771adb73 with delete_type TABLET_DATA_TOMBSTONED (TS d4cbc83202c8473d8c897582d30e5989 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42906
I20240503 14:16:23.184079 9689 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:23.189049 2561 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:23.192977 4468 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:23.204882 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f139d77b0f0c47809e6408b2771adb73 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:23.207453 2156 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 6 to 7, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:23.217442 4468 raft_consensus.cc:1467] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 14:16:23.218204 4468 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:16:23.289400 2144 catalog_manager.cc:4804] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): delete failed for tablet f139d77b0f0c47809e6408b2771adb73 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 5 but the committed config has opid_index of 7
I20240503 14:16:23.426935 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:23.435678 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:23.571907 9651 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:23.572407 9651 tablet_bootstrap.cc:492] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: Bootstrap complete.
I20240503 14:16:23.572732 9651 ts_tablet_manager.cc:1367] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent bootstrapping tablet: real 0.452s user 0.066s sys 0.000s
I20240503 14:16:23.573539 9651 raft_consensus.cc:348] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:23.573763 9651 raft_consensus.cc:729] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Initialized, Role: FOLLOWER
I20240503 14:16:23.573974 9651 consensus_queue.cc:260] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:23.580142 9651 ts_tablet_manager.cc:1398] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:16:23.583402 2320 tablet_copy_service.cc:342] P 7f22b4d5b1b5466dbbdd344f0721c05f: Request end of tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-944d2a983d6247498b02a6e7b7daf746 received from {username='slave'} at 127.20.99.8:46371
I20240503 14:16:23.583876 2320 tablet_copy_service.cc:433] P 7f22b4d5b1b5466dbbdd344f0721c05f: ending tablet copy session c2cf5d2244244a2db5ebc7dc46fd430d-944d2a983d6247498b02a6e7b7daf746 on tablet 944d2a983d6247498b02a6e7b7daf746 with peer c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:23.613107 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:23.620975 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:23.652837 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:23.677045 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:23.723289 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:23.748842 4579 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:23.752416 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:23.760212 2670 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:23.809659 2670 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:16:23.837354 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:23.840723 9667 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:23.841218 9667 tablet_bootstrap.cc:492] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: Bootstrap complete.
I20240503 14:16:23.841514 9667 ts_tablet_manager.cc:1367] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: Time spent bootstrapping tablet: real 0.714s user 0.061s sys 0.004s
I20240503 14:16:23.842310 9667 raft_consensus.cc:348] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:23.842550 9667 raft_consensus.cc:729] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7146de39e348442da4a363e95fbe0789, State: Initialized, Role: FOLLOWER
I20240503 14:16:23.842795 9667 consensus_queue.cc:260] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:23.848502 9667 ts_tablet_manager.cc:1398] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:16:23.849416 4619 tablet_copy_service.cc:342] P c2cf5d2244244a2db5ebc7dc46fd430d: Request end of tablet copy session 7146de39e348442da4a363e95fbe0789-1d584b7026b64ba1b720ed21d7019740 received from {username='slave'} at 127.20.99.5:52931
I20240503 14:16:23.849586 4619 tablet_copy_service.cc:433] P c2cf5d2244244a2db5ebc7dc46fd430d: ending tablet copy session 7146de39e348442da4a363e95fbe0789-1d584b7026b64ba1b720ed21d7019740 on tablet 1d584b7026b64ba1b720ed21d7019740 with peer 7146de39e348442da4a363e95fbe0789
I20240503 14:16:23.862130 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:23.876698 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:23.883787 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:23.892627 4202 raft_consensus.cc:1203] T 1d584b7026b64ba1b720ed21d7019740 P 7146de39e348442da4a363e95fbe0789 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:16:23.966194 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:23.979103 4598 raft_consensus.cc:1203] T 944d2a983d6247498b02a6e7b7daf746 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:16:23.999497 2690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59948
I20240503 14:16:23.999709 2690 raft_consensus.cc:595] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:24.056206 9803 raft_consensus.cc:979] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: : Instructing follower c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:24.087069 9803 raft_consensus.cc:1067] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Signalling peer c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:24.087893 4598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d"
dest_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
from {username='slave'} at 127.20.99.4:59775
I20240503 14:16:24.088112 4598 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:24.088229 4598 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:24.096275 4598 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:24.148056 4598 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:16:24.149981 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:24.150202 2560 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:24.150261 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:24.150467 2299 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:24.151371 2690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:24.151563 2690 raft_consensus.cc:3032] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:24.151620 2690 raft_consensus.cc:729] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:24.151868 2690 consensus_queue.cc:260] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:24.152387 2690 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:24.152980 2560 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:24.155089 2690 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:24.161846 2299 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:24.190784 2560 consensus_queue.cc:237] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:24.197160 4532 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c2cf5d2244244a2db5ebc7dc46fd430d, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:24.197723 9146 raft_consensus.cc:2781] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:24.249902 9146 raft_consensus.cc:686] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Becoming Leader. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Running, Role: LEADER
I20240503 14:16:24.259768 4333 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:24.260619 9790 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:24.275061 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
I20240503 14:16:24.275442 4468 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:24.281919 4468 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:24.283713 4468 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:24.284394 9790 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:24.296407 9146 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:24.298628 4202 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:24.299542 9790 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:24.321004 2430 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:24.321763 9790 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:24.341667 9790 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:24.343840 4201 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:24.347697 4468 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:24.352012 2430 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:24.354257 2158 catalog_manager.cc:5484] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:24.363821 4333 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:24.378357 2157 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: term changed from 1 to 2, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8). New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:24.394151 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:24.442886 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:24.453401 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:24.466753 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:24.493521 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:24.524410 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet aba6537cdb87400cb8b108b1fbfa6c0f with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:24.551203 2670 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:16:24.552294 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:24.562845 9881 tablet_replica.cc:330] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:24.563272 9881 raft_consensus.cc:2229] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:24.563652 9881 raft_consensus.cc:2258] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:24.566488 9881 ts_tablet_manager.cc:1875] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:24.587944 4449 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:24.621393 9881 ts_tablet_manager.cc:1888] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:24.621733 9881 log.cc:1196] T aba6537cdb87400cb8b108b1fbfa6c0f P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/aba6537cdb87400cb8b108b1fbfa6c0f
I20240503 14:16:24.642642 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet aba6537cdb87400cb8b108b1fbfa6c0f (table rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]) successfully deleted
I20240503 14:16:24.687027 2690 consensus_queue.cc:237] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:24.728731 2299 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:24.731204 9146 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:24.737497 4468 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:24.738138 9803 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:24.744478 2560 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:24.745309 9803 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:24.746804 2691 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:24.749799 9821 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:24.787369 9910 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:24.800347 2560 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:24.805128 4468 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:24.824282 2561 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:24.825035 9821 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:24.830030 4469 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:24.837970 9146 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:24.862845 2157 catalog_manager.cc:5484] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 4 to 5, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:24.867834 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet dd294a04366145378e3f5e3d306152e3 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:24.868218 9881 tablet_replica.cc:330] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:24.868976 9881 raft_consensus.cc:2229] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:24.869208 9881 raft_consensus.cc:2258] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:24.869959 9881 ts_tablet_manager.cc:1875] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:24.878204 9881 ts_tablet_manager.cc:1888] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:24.884670 9881 log.cc:1196] T dd294a04366145378e3f5e3d306152e3 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd294a04366145378e3f5e3d306152e3
I20240503 14:16:24.885807 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet dd294a04366145378e3f5e3d306152e3 (table rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]) successfully deleted
I20240503 14:16:25.020401 2691 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60006
I20240503 14:16:25.020670 2691 raft_consensus.cc:595] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:25.022151 2300 consensus_queue.cc:237] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:25.053592 4468 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:25.053808 4201 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:25.054445 9864 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:25.055271 9719 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:25.057440 2430 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:25.059993 9719 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:25.092587 9864 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:25.099234 4333 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:16:25.100633 9719 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:25.101073 2158 catalog_manager.cc:5484] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:25.110858 2670 tablet_service.cc:1514] Processing DeleteTablet for tablet af99824d1853471b867e4ff1f01e9367 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:25.098999 2430 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:25.121585 4469 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:25.129972 4201 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:25.159852 9972 tablet_replica.cc:330] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:25.160197 9972 raft_consensus.cc:2229] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:25.160763 9972 raft_consensus.cc:2258] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:25.161023 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:25.161657 9972 ts_tablet_manager.cc:1875] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:25.176471 4333 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:25.183418 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:25.190138 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:25.195844 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:25.212466 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:25.228879 9910 raft_consensus.cc:979] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: : Instructing follower d4cbc83202c8473d8c897582d30e5989 to start an election
I20240503 14:16:25.229074 9910 raft_consensus.cc:1067] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Signalling peer d4cbc83202c8473d8c897582d30e5989 to start an election
I20240503 14:16:25.229949 4468 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3"
dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
from {username='slave'} at 127.20.99.4:40293
I20240503 14:16:25.230238 4468 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:25.230389 4468 raft_consensus.cc:3037] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:25.236383 4468 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:25.238205 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:25.238472 2691 raft_consensus.cc:3032] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:25.238545 2691 raft_consensus.cc:729] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:25.238842 2691 consensus_queue.cc:260] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:25.239230 2691 raft_consensus.cc:3037] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:25.243218 2691 raft_consensus.cc:2454] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:25.244431 4402 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c37183c2965f46a28847c1262eb97829, d4cbc83202c8473d8c897582d30e5989; no voters:
I20240503 14:16:25.244742 9504 raft_consensus.cc:2781] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:25.246701 9504 raft_consensus.cc:686] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 2 LEADER]: Becoming Leader. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Running, Role: LEADER
I20240503 14:16:25.246995 9504 consensus_queue.cc:237] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:25.249862 2158 catalog_manager.cc:5484] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 reported cstate change: term changed from 1 to 2, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to d4cbc83202c8473d8c897582d30e5989 (127.20.99.7). New cstate: current_term: 2 leader_uuid: "d4cbc83202c8473d8c897582d30e5989" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:25.263547 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd294a04366145378e3f5e3d306152e3" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:25.263821 2561 raft_consensus.cc:3037] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:25.264099 9972 ts_tablet_manager.cc:1888] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:16:25.267053 9972 log.cc:1196] T af99824d1853471b867e4ff1f01e9367 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/af99824d1853471b867e4ff1f01e9367
I20240503 14:16:25.268200 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet af99824d1853471b867e4ff1f01e9367 (table rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]) successfully deleted
I20240503 14:16:25.269228 2561 raft_consensus.cc:2454] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4cbc83202c8473d8c897582d30e5989 in term 2.
I20240503 14:16:25.269943 4468 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 2 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:16:25.309145 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:25.367748 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:25.378610 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:25.633828 2561 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:25.634433 9504 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:25.695951 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:25.705793 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:25.747731 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:25.754899 4532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:44387, user_credentials={real_user=slave}} blocked reactor thread for 77481.2us
I20240503 14:16:25.797686 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:25.804803 2691 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:25.805577 9987 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:25.879856 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:25.917629 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:25.930703 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:25.998939 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.141006 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:26.144611 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:26.231956 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:26.236611 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.254774 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:26.260032 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:26.279685 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:26.283660 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:26.313804 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.323913 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.328768 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.382869 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.402082 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.406277 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:26.450060 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.455422 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:26.457284 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:26.473093 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:26.474191 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:26.553627 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:26.564596 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.583120 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:26.584332 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:26.609994 4598 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:26.626261 2561 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:26.627069 10048 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:26.630170 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:26.643671 4468 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:26.644217 2300 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:26.644234 10048 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:26.644896 10048 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:26.679520 10048 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:26.701377 2561 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:26.709024 2157 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:26.723636 2300 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:26.755344 4468 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:26.789623 2670 tablet_service.cc:1514] Processing DeleteTablet for tablet abdff0b719a342cfbde746751bd2a89d with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:26.795583 10174 tablet_replica.cc:330] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:26.795825 10174 raft_consensus.cc:2229] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:26.796007 10174 raft_consensus.cc:2258] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:26.798846 10174 ts_tablet_manager.cc:1875] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:26.805886 10174 ts_tablet_manager.cc:1888] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:26.806087 10174 log.cc:1196] T abdff0b719a342cfbde746751bd2a89d P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/abdff0b719a342cfbde746751bd2a89d
I20240503 14:16:26.806908 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet abdff0b719a342cfbde746751bd2a89d (table rebalance_test_table_6 [id=497c07d387f94011ae892d01054b8a2d]) successfully deleted
I20240503 14:16:26.881315 2430 consensus_queue.cc:237] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:26.909500 4598 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:26.910346 9494 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:26.928916 2691 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:26.928969 4468 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:26.929759 10191 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:26.930253 10191 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:26.951373 9494 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:26.957669 4468 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:26.967584 2691 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:26.968719 4598 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:26.977553 2151 catalog_manager.cc:5484] T dc88c5dc94844f1c8400c31ad1165a32 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:26.981540 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet dc88c5dc94844f1c8400c31ad1165a32 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:27.004819 10224 tablet_replica.cc:330] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:27.005964 10224 raft_consensus.cc:2229] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:27.007066 10224 raft_consensus.cc:2258] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:27.007920 10224 ts_tablet_manager.cc:1875] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:27.014397 10224 ts_tablet_manager.cc:1888] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:27.016868 10224 log.cc:1196] T dc88c5dc94844f1c8400c31ad1165a32 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc88c5dc94844f1c8400c31ad1165a32
I20240503 14:16:27.025553 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet dc88c5dc94844f1c8400c31ad1165a32 (table rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]) successfully deleted
I20240503 14:16:27.056649 2691 consensus_queue.cc:237] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:27.064188 4598 raft_consensus.cc:1261] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:27.064985 9911 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:27.078084 10087 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:27.078725 4597 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:27.085609 2157 catalog_manager.cc:5484] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 5 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:27.100327 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9974d80e8d31417788e10140de97e722 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:27.110572 2691 consensus_queue.cc:237] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:27.113685 4597 raft_consensus.cc:1261] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:16:27.113739 2627 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9974d80e8d31417788e10140de97e722. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:27.114356 10087 consensus_queue.cc:1035] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:27.116988 10087 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.119911 4597 raft_consensus.cc:2932] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.122875 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9974d80e8d31417788e10140de97e722 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:27.127599 2158 catalog_manager.cc:5484] T 9974d80e8d31417788e10140de97e722 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.151124 10261 tablet_replica.cc:330] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:27.153229 10261 raft_consensus.cc:2229] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:27.153601 10261 raft_consensus.cc:2258] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:27.155201 10261 ts_tablet_manager.cc:1875] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:27.183632 4469 consensus_queue.cc:237] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:27.190590 10261 ts_tablet_manager.cc:1888] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:27.190719 10261 log.cc:1196] T 9974d80e8d31417788e10140de97e722 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/9974d80e8d31417788e10140de97e722
I20240503 14:16:27.191521 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 9974d80e8d31417788e10140de97e722 (table rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]) successfully deleted
I20240503 14:16:27.198529 2561 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:27.199236 9504 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:27.202320 2691 consensus_queue.cc:237] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:27.236198 10279 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:27.238083 2561 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:27.241976 4201 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:27.244364 10087 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:27.255559 2157 catalog_manager.cc:5484] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "d4cbc83202c8473d8c897582d30e5989" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:27.268692 2670 tablet_service.cc:1514] Processing DeleteTablet for tablet dd294a04366145378e3f5e3d306152e3 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:27.269038 10174 tablet_replica.cc:330] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:27.269235 10174 raft_consensus.cc:2229] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:27.269424 10174 raft_consensus.cc:2258] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:27.270161 10174 ts_tablet_manager.cc:1875] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:27.278721 4597 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:27.279632 2430 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:27.280706 9911 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:27.281826 10087 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:27.284898 10174 ts_tablet_manager.cc:1888] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:27.284996 10174 log.cc:1196] T dd294a04366145378e3f5e3d306152e3 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/dd294a04366145378e3f5e3d306152e3
I20240503 14:16:27.285730 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet dd294a04366145378e3f5e3d306152e3 (table rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]) successfully deleted
I20240503 14:16:27.293339 10087 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.295323 4202 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
W20240503 14:16:27.360037 2628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 60526.1us
W20240503 14:16:27.361760 4402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:45269, user_credentials={real_user=slave}} blocked reactor thread for 118642us
I20240503 14:16:27.362875 2157 catalog_manager.cc:5484] T 832f07a6b12740d08fa67d653b3818bf P 7146de39e348442da4a363e95fbe0789 reported cstate change: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.363734 4469 consensus_queue.cc:237] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:27.366281 4597 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.369304 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 832f07a6b12740d08fa67d653b3818bf with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:27.369644 10224 tablet_replica.cc:330] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:27.369859 10224 raft_consensus.cc:2229] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:27.370066 10224 raft_consensus.cc:2258] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:16:27.370107 4404 consensus_peers.cc:485] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd294a04366145378e3f5e3d306152e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:27.370810 10224 ts_tablet_manager.cc:1875] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:27.373049 2430 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.380800 10224 ts_tablet_manager.cc:1888] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:27.381603 10224 log.cc:1196] T 832f07a6b12740d08fa67d653b3818bf P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/832f07a6b12740d08fa67d653b3818bf
I20240503 14:16:27.382385 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 832f07a6b12740d08fa67d653b3818bf (table rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]) successfully deleted
I20240503 14:16:27.393325 2561 raft_consensus.cc:1261] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer d4cbc83202c8473d8c897582d30e5989: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:27.396173 10288 consensus_queue.cc:1035] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:27.400084 9504 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.405429 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dd294a04366145378e3f5e3d306152e3 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:27.406167 2561 raft_consensus.cc:2932] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.413170 2158 catalog_manager.cc:5484] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) added. New cstate: current_term: 2 leader_uuid: "d4cbc83202c8473d8c897582d30e5989" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } }
I20240503 14:16:27.591410 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:27.620529 4533 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 90913.6us
I20240503 14:16:27.622638 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:27.622964 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:27.623716 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:27.627671 10352 ts_tablet_manager.cc:902] T 9974d80e8d31417788e10140de97e722 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:27.641000 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
F20240503 14:16:27.628087 10352 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-059a4a35aa94412994bb431342ba0753.queue_time_us, tablet-download-pool-059a4a35aa94412994bb431342ba0753.queue_time_us) (tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_cpu_time_us, tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_cpu_time_us) (tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_wall_time_us, tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_wall_time_us) (tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.queue_time_us, tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.queue_time_us) (tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_cpu_time_us, tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_cpu_time_us) (tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_wall_time_us, tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_wall_time_us) (tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.queue_time_us, tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.queue_time_us) (tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_cpu_time_us, tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_cpu_time_us) (tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_wall_time_us, tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_wall_time_us) (tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.queue_time_us, tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.queue_time_us) (tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_cpu_time_us, tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_cpu_time_us) (tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_wall_time_us, tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_wall_time_us) (tablet-download-pool-5f631c7713d24f4caf7550bb05147615.queue_time_us, tablet-download-pool-5f631c7713d24f4caf7550bb05147615.queue_time_us) (tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_cpu_time_us, tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_cpu_time_us) (tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_wall_time_us, tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_wall_time_us) (tablet-download-pool-61b63f87fb384bb2a48044bcad87c282.queue_time_us, tablet-download-pool-61b63f87fb384bb2a48044bcad87c282.queue_time_us) (tablet-download-pool-61b63f87fb384bb2a48044bcad87c282.run_cpu_time_us, tablet-download-pool-61b63f87fb384bb2a48044bcad87c282.run_cpu_time_us) (tablet-download-pool-61b63f87fb384bb2a48044bcad87c282.run_wall_time_us, tablet-download-pool-61b63f87fb384bb2a48044bcad87c282.run_wall_time_us) (tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.queue_time_us, tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.queue_time_us) (tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_cpu_time_us, tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_cpu_time_us) (tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_wall_time_us, tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_wall_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.queue_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.queue_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_cpu_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_cpu_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_wall_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_wall_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.queue_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.queue_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_cpu_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_cpu_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_wall_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_wall_time_us) (tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.queue_time_us, tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.queue_time_us) (tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_cpu_time_us, tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_cpu_time_us) (tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_wall_time_us, tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_wall_time_us) (tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.queue_time_us, tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.queue_time_us) (tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_cpu_time_us, tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_cpu_time_us) (tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_wall_time_us, tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_wall_time_us) (tablet-download-pool-9974d80e8d31417788e10140de97e722.queue_time_us, tablet-download-pool-9974d80e8d31417788e10140de97e722.queue_time_us) (tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.queue_time_us, tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.queue_time_us) (tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_cpu_time_us, tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_cpu_time_us) (tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_wall_time_us, tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_wall_time_us) (tablet-download-pool-af99824d1853471b867e4ff1f01e9367.queue_time_us, tablet-download-pool-af99824d1853471b867e4ff1f01e9367.queue_time_us) (tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_cpu_time_us, tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_cpu_time_us) (tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_wall_time_us, tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_wall_time_us) (tablet-download-pool-ff59573e2c8f472585b96ca0a0fd75b1.queue_time_us, tablet-download-pool-ff59573e2c8f472585b96ca0a0fd75b1.queue_time_us) (tablet-download-pool-ff59573e2c8f472585b96ca0a0fd75b1.run_cpu_time_us, tablet-download-pool-ff59573e2c8f472585b96ca0a0fd75b1.run_cpu_time_us) (tablet-download-pool-ff59573e2c8f472585b96ca0a0fd75b1.run_wall_time_us, tablet-download-pool-ff59573e2c8f472585b96ca0a0fd75b1.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fcf1304cdcd google::LogMessage::Fail() at ??:0
I20240503 14:16:27.675770 4183 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:27.682291 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7fcf13050b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fcf1304c7cc google::LogMessage::Flush() at ??:0
@ 0x7fcf1304df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcf15d8d742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fcf15d59f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fcf15d53cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fcf251a78de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 14:16:27.725211 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7fcf251bfdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fcf252ea480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fcf25322a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fcf25305981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fcf2530570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fcf15d5db75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcf15d6119c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 14:16:27.851792 10393 ts_tablet_manager.cc:902] T dd294a04366145378e3f5e3d306152e3 P 7146de39e348442da4a363e95fbe0789: Initiating tablet copy from peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
@ 0x7fcf15d61019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fcf15d3c01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcf190c16db start_thread at ??:0
@ 0x7fcf11eff71f clone at ??:0
I20240503 14:16:27.961829 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:27.972429 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:28.004166 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.008527 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.009620 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.039809 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:28.071077 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:28.380007 2366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 82699.1us
I20240503 14:16:28.392475 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:28.406493 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:28.416445 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.444334 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:28.447698 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:28.500180 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.521996 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.635504 4579 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.636407 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.658643 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:28.660583 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:28.681267 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:28.683641 2670 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:28.714917 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:28.847867 2627 connection.cc:535] client connection to 127.20.99.5:41737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.848135 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.849809 4404 connection.cc:535] client connection to 127.20.99.5:41737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.850090 4404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.850586 2236 connection.cc:535] client connection to 127.20.99.5:41737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.850803 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.853605 2497 connection.cc:535] client connection to 127.20.99.5:41737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.853854 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.859784 2367 connection.cc:535] client connection to 127.20.99.5:41737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.860055 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.864763 4534 connection.cc:535] client connection to 127.20.99.5:41737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.865000 4534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.887881 4534 consensus_peers.cc:485] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.889925 4534 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:28.893900 2300 consensus_queue.cc:237] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
W20240503 14:16:28.898411 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.898547 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.906643 10527 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:28.852407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.5:41737 (local address 127.20.99.4:48911)
0503 14:16:28.891012 (+ 38605us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:28.891030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:28.891153 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:28.905554 (+ 14401us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38505,"thread_start_us":38419,"threads_started":1}
W20240503 14:16:28.906875 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.906982 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.907054 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.907143 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.909832 10531 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:28.861787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.5:41737 (local address 127.20.99.7:33543)
0503 14:16:28.893299 (+ 31512us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:28.893318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:28.893435 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:28.908902 (+ 15467us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31402,"thread_start_us":38381,"threads_started":1}
W20240503 14:16:28.910044 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:28.910192 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.918293 2367 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 58230.8us
W20240503 14:16:28.930840 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.931005 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.931094 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.931541 2497 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 77746.9us
W20240503 14:16:28.933714 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.933846 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:28.933933 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:28.940532 4333 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:28.941138 2691 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:28.941383 9960 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:28.941988 9960 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:28.969592 10549 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:28.979477 2691 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:28.985911 4333 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:28.991672 2157 catalog_manager.cc:5484] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:29.007786 2691 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45956
I20240503 14:16:29.008001 2691 raft_consensus.cc:595] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:29.022888 10566 raft_consensus.cc:979] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: : Instructing follower c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:29.023133 10312 raft_consensus.cc:1067] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Signalling peer c2cf5d2244244a2db5ebc7dc46fd430d to start an election
I20240503 14:16:29.023919 4597 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57"
dest_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d"
from {username='slave'} at 127.20.99.4:59775
I20240503 14:16:29.024186 4597 raft_consensus.cc:482] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:29.024296 4597 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:29.027510 4597 raft_consensus.cc:504] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:29.028956 4597 leader_election.cc:288] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:29.029346 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:29.029563 2561 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:29.032933 2561 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:29.033419 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:29.033634 2300 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:29.037451 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
I20240503 14:16:29.037706 4333 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:29.041188 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:29.041401 2691 raft_consensus.cc:3032] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:29.041476 2691 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:29.041754 2691 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:29.042243 2691 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:29.045604 4333 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:29.046056 4535 leader_election.cc:302] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, b35c386497d046de9be5007a31ab7d10, c2cf5d2244244a2db5ebc7dc46fd430d; no voters:
I20240503 14:16:29.046401 10048 raft_consensus.cc:2781] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:29.056053 2300 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:29.057636 2691 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cf5d2244244a2db5ebc7dc46fd430d in term 2.
I20240503 14:16:29.092079 10048 raft_consensus.cc:686] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Becoming Leader. State: Replica: c2cf5d2244244a2db5ebc7dc46fd430d, State: Running, Role: LEADER
I20240503 14:16:29.092490 10048 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:29.098783 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae190d0f4d843c990e6fdadcac677ed with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:29.104251 2157 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: term changed from 1 to 2, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8). New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:29.131196 10589 tablet_replica.cc:330] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:29.131417 10589 raft_consensus.cc:2229] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:29.131610 10589 raft_consensus.cc:2258] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:16:29.140117 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:29.142320 10589 ts_tablet_manager.cc:1875] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:29.156852 10589 ts_tablet_manager.cc:1888] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:29.156993 10589 log.cc:1196] T 9ae190d0f4d843c990e6fdadcac677ed P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/9ae190d0f4d843c990e6fdadcac677ed
I20240503 14:16:29.157756 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 9ae190d0f4d843c990e6fdadcac677ed (table rebalance_test_table_8 [id=b0dd533aba93443fb745a864d9511793]) successfully deleted
W20240503 14:16:29.179584 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:29.278585 2430 consensus_queue.cc:237] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:29.287247 4333 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:29.288033 10194 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:29.292966 4598 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:29.293774 10194 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:29.300410 2561 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:29.302585 10460 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:29.365151 10194 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:29.369005 4597 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:29.388516 2157 catalog_manager.cc:5484] T 6c94404d8d4f45d59958ae6feb36fc24 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 5 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:29.411612 2561 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:29.421782 4333 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:29.426806 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c94404d8d4f45d59958ae6feb36fc24 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:29.443526 10643 tablet_replica.cc:330] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:29.443922 10643 raft_consensus.cc:2229] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:29.444272 10643 raft_consensus.cc:2258] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:29.445219 10643 ts_tablet_manager.cc:1875] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:29.452728 10643 ts_tablet_manager.cc:1888] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:16:29.452932 10643 log.cc:1196] T 6c94404d8d4f45d59958ae6feb36fc24 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c94404d8d4f45d59958ae6feb36fc24
I20240503 14:16:29.453836 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet 6c94404d8d4f45d59958ae6feb36fc24 (table rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]) successfully deleted
I20240503 14:16:29.516774 2561 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 14:16:29.517083 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:29.517617 10048 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:29.632928 4333 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:29.633713 10048 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:29.636621 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:29.681622 2300 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:29.682750 2691 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:29.683890 10048 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:29.684589 10048 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:29.796633 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:29.801656 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:29.847079 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:29.894320 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:29.964308 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:29.965678 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:29.979566 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:30.025454 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.044744 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:30.045990 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:30.077704 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.080626 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:30.100102 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:30.113175 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:30.140352 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:30.235242 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:16:30.251022 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:30.277012 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.295519 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.329386 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:30.340343 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:30.371052 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:30.371516 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:30.421988 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:30.438550 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:30.529137 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.531569 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.533165 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:30.557742 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:30.558319 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:30.598347 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:16:30.660053 2234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.7:44387, user_credentials={real_user=slave}} blocked reactor thread for 70995.8us
W20240503 14:16:30.741577 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:30.824007 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:30.852417 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:30.881848 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:30.909250 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:30.942402 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:16:30.966797 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.009418 2300 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d"
dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50424
I20240503 14:16:31.009665 2300 raft_consensus.cc:595] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:16:31.029047 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:31.040805 2280 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:31.043748 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:16:31.058912 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:31.060940 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:31.068493 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.103832 4598 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:16:31.116930 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.119674 4334 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:31.120795 2561 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:31.121285 2300 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:31.121929 10697 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:31.129101 10558 raft_consensus.cc:979] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: : Instructing follower 685c6181561b4056abdecfd6d8bdb3b2 to start an election
I20240503 14:16:31.129225 10558 raft_consensus.cc:1067] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Signalling peer 685c6181561b4056abdecfd6d8bdb3b2 to start an election
I20240503 14:16:31.129915 2430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d"
dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
from {username='slave'} at 127.20.99.1:55961
I20240503 14:16:31.130079 2430 raft_consensus.cc:482] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:31.130151 2430 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.132820 10697 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:31.133358 10697 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:31.138796 2430 raft_consensus.cc:504] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:31.140130 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:31.140349 2300 raft_consensus.cc:3032] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:31.140424 2300 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Running, Role: LEADER
I20240503 14:16:31.140702 2300 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:31.141096 2300 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.144786 2300 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:31.151223 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
I20240503 14:16:31.151420 4468 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.151612 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4675882e64f35bc3dc9ee06ae366d" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:31.151852 2691 raft_consensus.cc:3037] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.154153 4468 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:31.154489 2691 raft_consensus.cc:2454] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:31.154654 2365 leader_election.cc:302] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, 7f22b4d5b1b5466dbbdd344f0721c05f, d4cbc83202c8473d8c897582d30e5989; no voters:
I20240503 14:16:31.155009 10785 raft_consensus.cc:2781] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:31.155340 10785 raft_consensus.cc:686] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:16:31.155612 10785 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:31.157948 2430 leader_election.cc:288] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:16:31.159206 2157 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 1 to 2, leader changed from 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:31.205767 2430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f"
dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43018
I20240503 14:16:31.205966 2430 raft_consensus.cc:595] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:16:31.218904 4535 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 95802.1us
W20240503 14:16:31.229979 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:31.232519 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.245098 10697 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:31.252645 2561 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:31.254807 4334 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:31.265621 10785 raft_consensus.cc:979] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: : Instructing follower b35c386497d046de9be5007a31ab7d10 to start an election
I20240503 14:16:31.266098 10194 raft_consensus.cc:1067] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Signalling peer b35c386497d046de9be5007a31ab7d10 to start an election
I20240503 14:16:31.267223 2561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f"
dest_uuid: "b35c386497d046de9be5007a31ab7d10"
from {username='slave'} at 127.20.99.2:53035
I20240503 14:16:31.267505 2561 raft_consensus.cc:482] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:31.267616 2561 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.269088 2300 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:31.278079 2153 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:31.283695 2561 raft_consensus.cc:504] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:31.285228 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:31.285509 2300 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.287272 2670 tablet_service.cc:1514] Processing DeleteTablet for tablet fda4460865854505b5220d7ec3b34c57 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:31.291707 2300 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
I20240503 14:16:31.292919 2561 leader_election.cc:288] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 2 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:31.293396 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
I20240503 14:16:31.293598 4334 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.296295 4334 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
I20240503 14:16:31.296854 2498 leader_election.cc:302] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:16:31.297235 10166 raft_consensus.cc:2781] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:31.298048 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a474791f9b4585aa0114162705c96f" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:16:31.298271 2430 raft_consensus.cc:3032] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:31.298352 2430 raft_consensus.cc:729] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:16:31.298625 2430 consensus_queue.cc:260] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:31.299098 2430 raft_consensus.cc:3037] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:31.301750 2430 raft_consensus.cc:2454] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
W20240503 14:16:31.326942 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:31.327984 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:31.335364 10166 raft_consensus.cc:686] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:16:31.336254 10166 consensus_queue.cc:237] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:31.353032 10919 tablet_replica.cc:330] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:31.353263 10919 raft_consensus.cc:2229] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:31.354220 10919 raft_consensus.cc:2258] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:31.355263 10919 ts_tablet_manager.cc:1875] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:16:31.356001 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.358351 2159 catalog_manager.cc:5484] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 1 to 2, leader changed from 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:16:31.389315 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:31.391971 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.413381 10919 ts_tablet_manager.cc:1888] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:31.416863 10919 log.cc:1196] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/fda4460865854505b5220d7ec3b34c57
I20240503 14:16:31.417984 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet fda4460865854505b5220d7ec3b34c57 (table rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]) successfully deleted
W20240503 14:16:31.430261 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:31.554920 4534 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:31.556567 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:31.577828 2300 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:31.578378 10915 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:31.610980 2691 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:31.612017 10915 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:31.685811 4469 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:31.686398 10785 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:31.708938 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:31.728937 2300 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:31.729642 10921 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:31.730599 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:31.740195 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:16:31.778453 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:31.780190 4534 consensus_peers.cc:485] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:31.827345 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:31.853667 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:31.858022 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:31.897716 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:31.930503 2430 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:31.952248 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:31.982107 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:31.986761 2692 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:31.990172 11045 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:32.004455 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:32.006651 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:32.007647 4334 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:16:32.008160 2495 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 78320.1us
I20240503 14:16:32.008247 10869 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:32.012159 4333 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:16:32.012751 10921 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:32.024164 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:32.025951 2561 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:32.029731 11054 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:16:32.062503 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 66270us
I20240503 14:16:32.064034 10869 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.068231 4334 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.079552 2561 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.080039 2156 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 4 to 5, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.088822 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:32.122902 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:32.141523 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:32.156297 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:32.159929 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1d30b76c4d43d0a54a49892d09a6f8 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:32.179003 11075 tablet_replica.cc:330] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:32.179380 11075 raft_consensus.cc:2229] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:32.181754 11075 raft_consensus.cc:2258] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:32.184852 11075 ts_tablet_manager.cc:1875] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:32.189183 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:32.190872 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:32.196974 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:32.218098 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:32.236179 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:32.240813 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:32.253343 11075 ts_tablet_manager.cc:1888] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:16:32.253577 11075 log.cc:1196] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb1d30b76c4d43d0a54a49892d09a6f8
I20240503 14:16:32.254871 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet eb1d30b76c4d43d0a54a49892d09a6f8 (table rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]) successfully deleted
W20240503 14:16:32.391371 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:32.537488 4597 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:32.556838 2561 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:32.557039 4334 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer c2cf5d2244244a2db5ebc7dc46fd430d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:32.557372 10890 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:32.557902 10890 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:32.581488 10890 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:32.590504 2156 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P c2cf5d2244244a2db5ebc7dc46fd430d reported cstate change: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:32.597857 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet fda4460865854505b5220d7ec3b34c57 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:32.603477 4334 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:32.603677 2561 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:16:32.631858 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:32.633713 2430 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:32.636547 4469 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:32.637859 11043 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:32.638754 2691 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:32.639577 10915 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:32.644798 4469 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:32.643049 11043 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:32.651855 2691 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:32.664726 11150 tablet_replica.cc:330] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:32.665191 11150 raft_consensus.cc:2229] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:32.665664 11150 raft_consensus.cc:2258] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:32.666625 11150 ts_tablet_manager.cc:1875] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:32.667050 2156 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 4 to 6, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:32.677172 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf4675882e64f35bc3dc9ee06ae366d with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:32.693169 11150 ts_tablet_manager.cc:1888] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:16:32.714766 11150 log.cc:1196] T fda4460865854505b5220d7ec3b34c57 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/fda4460865854505b5220d7ec3b34c57
I20240503 14:16:32.717864 11150 tablet_replica.cc:330] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:32.718150 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet fda4460865854505b5220d7ec3b34c57 (table rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]) successfully deleted
I20240503 14:16:32.719365 11150 raft_consensus.cc:2229] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:32.719822 11150 raft_consensus.cc:2258] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:32.720665 11150 ts_tablet_manager.cc:1875] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:32.760608 11150 ts_tablet_manager.cc:1888] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:16:32.762195 11150 log.cc:1196] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:16:32.802282 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet 8bf4675882e64f35bc3dc9ee06ae366d (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
W20240503 14:16:32.850972 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:32.894425 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:32.912700 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:32.928634 2561 consensus_queue.cc:237] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:32.931136 2301 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:32.932118 11045 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:32.933640 4334 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:32.934679 11045 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:32.935609 11045 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.941228 4334 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.946924 2151 catalog_manager.cc:5484] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 4 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.946960 2301 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:32.953112 2412 tablet_service.cc:1514] Processing DeleteTablet for tablet a9a474791f9b4585aa0114162705c96f with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39122
I20240503 14:16:32.964709 11239 tablet_replica.cc:330] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:16:32.965018 11239 raft_consensus.cc:2229] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:32.965296 11239 raft_consensus.cc:2258] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:32.965708 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:32.980651 11239 ts_tablet_manager.cc:1875] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:32.985276 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:32.987044 11239 ts_tablet_manager.cc:1888] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:16:33.007630 11239 log.cc:1196] T a9a474791f9b4585aa0114162705c96f P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/a9a474791f9b4585aa0114162705c96f
I20240503 14:16:33.008571 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet a9a474791f9b4585aa0114162705c96f (table rebalance_test_table_7 [id=37ac6a3a785d4dfead9698313d2651c9]) successfully deleted
W20240503 14:16:33.017560 2498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 71827.3us
W20240503 14:16:33.029278 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:33.031920 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:33.041425 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:33.091172 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:33.208102 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:33.255826 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:33.278246 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:16:33.342885 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:33.358543 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:33.374009 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:33.374433 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:33.409320 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:33.413044 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:33.521620 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:33.550769 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:33.568477 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:33.593941 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:33.604250 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 14:16:33.621801 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:33.647567 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:33.661365 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:33.673970 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:33.710554 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:33.710742 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:33.715221 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:33.721668 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:33.781419 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:33.815938 4534 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:33.839864 4334 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
W20240503 14:16:33.841934 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:33.862007 4469 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:33.862242 2300 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:33.862670 8187 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:33.863209 8187 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:33.868026 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:16:33.869123 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:33.876598 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:33.884651 11359 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:33.888861 4468 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:33.898458 2159 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 7 to 8, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:33.953513 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet b359fb8d9afc4c9c856897a722100304 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35800
I20240503 14:16:33.955147 2301 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:16:34.007679 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:16:34.008215 11383 tablet_replica.cc:330] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:34.008522 11383 raft_consensus.cc:2229] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Raft consensus shutting down.
W20240503 14:16:34.008867 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:34.009193 11383 raft_consensus.cc:2258] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:34.010160 11383 ts_tablet_manager.cc:1875] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:34.037135 2691 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46122
I20240503 14:16:34.037421 2691 raft_consensus.cc:595] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:16:34.080371 2412 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:34.083556 11383 ts_tablet_manager.cc:1888] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:16:34.086978 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:34.091548 11383 log.cc:1196] T b359fb8d9afc4c9c856897a722100304 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/b359fb8d9afc4c9c856897a722100304
I20240503 14:16:34.092928 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet b359fb8d9afc4c9c856897a722100304 (table rebalance_test_table_5 [id=39fac270fc164b88879eae47f2f4feb9]) successfully deleted
W20240503 14:16:34.109423 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:34.141469 4534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:16:34.142654 4534 consensus_peers.cc:485] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:34.146857 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:34.149310 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:34.149325 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:34.149920 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:34.161064 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:16:34.167322 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 14:16:34.191686 4404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:16:34.196072 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:34.202775 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:34.222254 10869 raft_consensus.cc:979] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: : Instructing follower 685c6181561b4056abdecfd6d8bdb3b2 to start an election
I20240503 14:16:34.222474 10869 raft_consensus.cc:1067] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 LEADER]: Signalling peer 685c6181561b4056abdecfd6d8bdb3b2 to start an election
I20240503 14:16:34.223228 2430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36"
dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
from {username='slave'} at 127.20.99.4:45821
I20240503 14:16:34.223464 2430 raft_consensus.cc:482] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:34.223574 2430 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:34.226155 2670 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:16:34.228160 2430 raft_consensus.cc:504] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:34.230469 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:34.231285 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75bafc7675ca45639b696db6e4b7ce36" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
I20240503 14:16:34.231468 4468 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:34.234148 4468 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 3.
I20240503 14:16:34.234786 2365 leader_election.cc:302] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, d4cbc83202c8473d8c897582d30e5989; no voters:
I20240503 14:16:34.235253 10915 raft_consensus.cc:2781] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:34.235407 10915 raft_consensus.cc:686] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 LEADER]: Becoming Leader. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Running, Role: LEADER
I20240503 14:16:34.235677 10915 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:34.241122 2430 leader_election.cc:288] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 3 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387)
I20240503 14:16:34.249836 2151 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: term changed from 2 to 3, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2). New cstate: current_term: 3 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:34.250900 2692 raft_consensus.cc:3032] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:34.250980 2692 raft_consensus.cc:729] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:34.251235 2692 consensus_queue.cc:260] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:34.251593 2692 raft_consensus.cc:3037] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:34.254320 2692 raft_consensus.cc:2454] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 3.
W20240503 14:16:34.274298 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:34.295974 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:34.323269 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:34.339401 4315 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:34.340860 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:34.374883 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:34.384851 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:34.425403 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:34.617254 2692 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 14:16:34.617977 10915 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:34.653877 2561 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754"
dest_uuid: "b35c386497d046de9be5007a31ab7d10"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55066
I20240503 14:16:34.654065 2561 raft_consensus.cc:595] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:16:34.678560 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:34.684867 4468 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240503 14:16:34.685637 11161 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:34.726985 2430 consensus_queue.cc:237] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
W20240503 14:16:34.746938 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:34.748325 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 470b425f5cef494ab3334fe2b85e1c4e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:34.749330 2300 raft_consensus.cc:1261] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:34.749949 10915 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:34.754137 2561 raft_consensus.cc:1261] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:16:34.754786 10915 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:34.774853 10915 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:34.779568 2301 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:34.796651 2151 catalog_manager.cc:5484] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:34.806619 11493 raft_consensus.cc:979] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: : Instructing follower 285488446ac445c5a5dfe654bfaf9835 to start an election
I20240503 14:16:34.806739 11493 raft_consensus.cc:1067] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Signalling peer 285488446ac445c5a5dfe654bfaf9835 to start an election
I20240503 14:16:34.807463 4334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754"
dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
from {username='slave'} at 127.20.99.3:46253
I20240503 14:16:34.807655 4334 raft_consensus.cc:482] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:34.807752 4334 raft_consensus.cc:3037] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:34.810607 4334 raft_consensus.cc:504] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:34.811928 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:34.812110 2561 raft_consensus.cc:3032] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:34.812172 2561 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:16:34.812417 2561 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:34.812803 2561 raft_consensus.cc:3037] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:34.816635 4334 leader_election.cc:288] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 2 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:34.817246 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32927a23842b4725b9dfafbac47fe754" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:34.817497 2691 raft_consensus.cc:3037] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:34.820628 2691 raft_consensus.cc:2454] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:34.821142 4268 leader_election.cc:302] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:34.821410 11455 raft_consensus.cc:2781] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:34.824765 2561 raft_consensus.cc:2454] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:34.834204 2561 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:34.841562 11455 raft_consensus.cc:686] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Becoming Leader. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Running, Role: LEADER
I20240503 14:16:34.842020 11455 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
W20240503 14:16:34.878198 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:34.884929 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:34.894100 2159 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: term changed from 1 to 2, leader changed from b35c386497d046de9be5007a31ab7d10 (127.20.99.3) to 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6). New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:34.895784 4315 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:34.907958 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:34.913627 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:34.938041 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:34.947914 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:35.022938 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:35.101372 4315 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:35.112077 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:35.138960 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:35.154352 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:35.156071 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:35.163079 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:35.173918 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:35.212687 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:35.229158 2561 consensus_queue.cc:237] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:16:35.244767 2498 consensus_peers.cc:485] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9a474791f9b4585aa0114162705c96f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:35.245661 4333 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:35.246109 11387 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:35.250563 11600 ts_tablet_manager.cc:902] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:35.252414 2300 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:35.252961 11387 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:35.286604 11387 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.291286 4334 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.298990 2151 catalog_manager.cc:5484] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.300233 2300 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.355890 11600 tablet_copy_client.cc:308] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:35.356601 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 470b425f5cef494ab3334fe2b85e1c4e from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:49643)
I20240503 14:16:35.356770 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 470b425f5cef494ab3334fe2b85e1c4e from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:49643: session id = d4cbc83202c8473d8c897582d30e5989-470b425f5cef494ab3334fe2b85e1c4e
I20240503 14:16:35.360898 2452 tablet_copy_source_session.cc:205] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:35.362265 11600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470b425f5cef494ab3334fe2b85e1c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:35.368782 2692 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:35.369350 11455 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:35.373732 11600 tablet_copy_client.cc:783] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:35.374527 11600 tablet_copy_client.cc:647] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:35.376775 11600 tablet_copy_client.cc:523] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:35.381536 11600 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
W20240503 14:16:35.401659 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:35.415552 2561 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:35.416334 11455 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:35.562081 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:35.581138 11600 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:35.581790 11600 tablet_bootstrap.cc:492] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:35.582311 11600 ts_tablet_manager.cc:1367] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.201s user 0.075s sys 0.077s
I20240503 14:16:35.583470 11600 raft_consensus.cc:348] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:35.583988 11600 raft_consensus.cc:729] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:35.584414 11600 consensus_queue.cc:260] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:35.585326 11600 ts_tablet_manager.cc:1398] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:35.586077 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-470b425f5cef494ab3334fe2b85e1c4e received from {username='slave'} at 127.20.99.7:49643
I20240503 14:16:35.586216 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-470b425f5cef494ab3334fe2b85e1c4e on tablet 470b425f5cef494ab3334fe2b85e1c4e with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:35.620361 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:35.642772 2300 consensus_queue.cc:237] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:35.667027 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:35.689231 4315 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:35.703222 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:35.704533 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:35.729226 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:35.764070 11696 ts_tablet_manager.cc:902] T a9a474791f9b4585aa0114162705c96f P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
W20240503 14:16:35.765836 2237 consensus_peers.cc:485] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ae190d0f4d843c990e6fdadcac677ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:35.770216 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:35.799574 2430 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:35.801054 4333 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:35.801862 11442 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:35.811058 2692 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:16:35.825223 4469 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 14:16:35.825822 11529 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:35.831202 11442 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:35.834079 11687 log.cc:924] Time spent T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
F20240503 14:16:35.829910 11696 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-059a4a35aa94412994bb431342ba0753.queue_time_us, tablet-download-pool-059a4a35aa94412994bb431342ba0753.queue_time_us) (tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_cpu_time_us, tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_cpu_time_us) (tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_wall_time_us, tablet-download-pool-059a4a35aa94412994bb431342ba0753.run_wall_time_us) (tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.queue_time_us, tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.queue_time_us) (tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_cpu_time_us, tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_cpu_time_us) (tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_wall_time_us, tablet-download-pool-1d584b7026b64ba1b720ed21d7019740.run_wall_time_us) (tablet-download-pool-5f631c7713d24f4caf7550bb05147615.queue_time_us, tablet-download-pool-5f631c7713d24f4caf7550bb05147615.queue_time_us) (tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_cpu_time_us, tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_cpu_time_us) (tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_wall_time_us, tablet-download-pool-5f631c7713d24f4caf7550bb05147615.run_wall_time_us) (tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.queue_time_us, tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.queue_time_us) (tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_cpu_time_us, tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_cpu_time_us) (tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_wall_time_us, tablet-download-pool-6325a400641b4d25b95592abb4af7d2d.run_wall_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.queue_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.queue_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_cpu_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_cpu_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_wall_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_wall_time_us) (tablet-download-pool-6c94404d8d4f45d59958ae6feb36fc24.queue_time_us, tablet-download-pool-6c94404d8d4f45d59958ae6feb36fc24.queue_time_us) (tablet-download-pool-6c94404d8d4f45d59958ae6feb36fc24.run_cpu_time_us, tablet-download-pool-6c94404d8d4f45d59958ae6feb36fc24.run_cpu_time_us) (tablet-download-pool-6c94404d8d4f45d59958ae6feb36fc24.run_wall_time_us, tablet-download-pool-6c94404d8d4f45d59958ae6feb36fc24.run_wall_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.queue_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.queue_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_cpu_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_cpu_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_wall_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_wall_time_us) (tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.queue_time_us, tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.queue_time_us) (tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_cpu_time_us, tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_cpu_time_us) (tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_wall_time_us, tablet-download-pool-832f07a6b12740d08fa67d653b3818bf.run_wall_time_us) (tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.queue_time_us, tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.queue_time_us) (tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_cpu_time_us, tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_cpu_time_us) (tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_wall_time_us, tablet-download-pool-944d2a983d6247498b02a6e7b7daf746.run_wall_time_us) (tablet-download-pool-9974d80e8d31417788e10140de97e722.queue_time_us, tablet-download-pool-9974d80e8d31417788e10140de97e722.queue_time_us) (tablet-download-pool-9974d80e8d31417788e10140de97e722.run_cpu_time_us, tablet-download-pool-9974d80e8d31417788e10140de97e722.run_cpu_time_us) (tablet-download-pool-9974d80e8d31417788e10140de97e722.run_wall_time_us, tablet-download-pool-9974d80e8d31417788e10140de97e722.run_wall_time_us) (tablet-download-pool-a9a474791f9b4585aa0114162705c96f.queue_time_us, tablet-download-pool-a9a474791f9b4585aa0114162705c96f.queue_time_us) (tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.queue_time_us, tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.queue_time_us) (tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_cpu_time_us, tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_cpu_time_us) (tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_wall_time_us, tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_wall_time_us) (tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.queue_time_us, tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.queue_time_us) (tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_cpu_time_us, tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_cpu_time_us) (tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_wall_time_us, tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_wall_time_us) (tablet-download-pool-fda4460865854505b5220d7ec3b34c57.queue_time_us, tablet-download-pool-fda4460865854505b5220d7ec3b34c57.queue_time_us) (tablet-download-pool-fda4460865854505b5220d7ec3b34c57.run_cpu_time_us, tablet-download-pool-fda4460865854505b5220d7ec3b34c57.run_cpu_time_us) (tablet-download-pool-fda4460865854505b5220d7ec3b34c57.run_wall_time_us, tablet-download-pool-fda4460865854505b5220d7ec3b34c57.run_wall_time_us) (tablet-download-pool-fe0eebf47c494dad8edf06197cdf15bc.queue_time_us, tablet-download-pool-fe0eebf47c494dad8edf06197cdf15bc.queue_time_us) (tablet-download-pool-fe0eebf47c494dad8edf06197cdf15bc.run_cpu_time_us, tablet-download-pool-fe0eebf47c494dad8edf06197cdf15bc.run_cpu_time_us) (tablet-download-pool-fe0eebf47c494dad8edf06197cdf15bc.run_wall_time_us, tablet-download-pool-fe0eebf47c494dad8edf06197cdf15bc.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 14:16:35.836093 11529 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
*** Check failure stack trace: ***
I20240503 14:16:35.845317 4468 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:35.852842 2151 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 6 to 8, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 3 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
@ 0x7f43f0d6bdcd google::LogMessage::Fail() at ??:0
I20240503 14:16:35.865428 4315 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
@ 0x7f43f0d6fb93 google::LogMessage::SendToLog() at ??:0
W20240503 14:16:35.869284 2234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 57234.1us
I20240503 14:16:35.884449 11711 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.888221 2692 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.899560 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:35.900159 2670 tablet_service.cc:1514] Processing DeleteTablet for tablet 75bafc7675ca45639b696db6e4b7ce36 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:35.900976 11735 tablet_replica.cc:330] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:35.901161 11735 raft_consensus.cc:2229] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:35.901335 11735 raft_consensus.cc:2258] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:35.901912 2430 consensus_queue.cc:237] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:16:35.903959 2368 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75bafc7675ca45639b696db6e4b7ce36. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:35.907959 4469 raft_consensus.cc:1261] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 14:16:35.909108 11529 consensus_queue.cc:1035] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:35.910516 4579 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:35.921276 4334 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.923781 11529 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.929149 2153 catalog_manager.cc:5484] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 5 to 6, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.929525 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:35.932186 2145 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 75bafc7675ca45639b696db6e4b7ce36 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:35.933570 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:35.933929 4469 raft_consensus.cc:2932] T 75bafc7675ca45639b696db6e4b7ce36 P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:35.936401 2151 catalog_manager.cc:5484] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 8 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) added. New cstate: current_term: 3 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
@ 0x7f43f0d6b7cc google::LogMessage::Flush() at ??:0
I20240503 14:16:35.949359 11735 ts_tablet_manager.cc:1875] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7f43f0d6cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 14:16:35.997181 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:36.008036 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:36.008090 11735 ts_tablet_manager.cc:1888] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240503 14:16:36.008203 11735 log.cc:1196] T 75bafc7675ca45639b696db6e4b7ce36 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/75bafc7675ca45639b696db6e4b7ce36
I20240503 14:16:36.008950 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet 75bafc7675ca45639b696db6e4b7ce36 (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
I20240503 14:16:36.046758 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:36.048523 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7f43f3aac742 kudu::TraceMetrics::InternName() at ??:0
W20240503 14:16:36.100657 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:36.121696 2692 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
W20240503 14:16:36.127135 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:36.148183 2562 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:16:36.149390 11773 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:36.153131 4469 raft_consensus.cc:1203] T 470b425f5cef494ab3334fe2b85e1c4e P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:16:36.173790 11775 ts_tablet_manager.cc:902] T 9ae190d0f4d843c990e6fdadcac677ed P c2cf5d2244244a2db5ebc7dc46fd430d: Initiating tablet copy from peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597)
@ 0x7f43f3a78f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 14:16:36.189950 11773 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:36.192598 2562 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:36.200569 2153 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 7 to 8, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:36.205861 4449 tablet_service.cc:1514] Processing DeleteTablet for tablet f139d77b0f0c47809e6408b2771adb73 with delete_type TABLET_DATA_TOMBSTONED (TS d4cbc83202c8473d8c897582d30e5989 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:42906
I20240503 14:16:36.206775 2692 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
W20240503 14:16:36.209786 2626 consensus_peers.cc:485] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 -> Peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): Couldn't send request to peer 7f22b4d5b1b5466dbbdd344f0721c05f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:36.225463 2562 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:16:36.226048 11773 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:36.227358 11783 tablet_replica.cc:330] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: stopping tablet replica
I20240503 14:16:36.227906 11783 raft_consensus.cc:2229] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:36.228188 11783 raft_consensus.cc:2258] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:36.228894 11423 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:36.229571 11783 ts_tablet_manager.cc:1875] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:36.229471 2561 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:36.235277 2153 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 8 to 9, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:36.242509 11783 ts_tablet_manager.cc:1888] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:16:36.242642 11783 log.cc:1196] T f139d77b0f0c47809e6408b2771adb73 P d4cbc83202c8473d8c897582d30e5989: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-6/wal/wals/f139d77b0f0c47809e6408b2771adb73
I20240503 14:16:36.244534 2144 catalog_manager.cc:4830] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): tablet f139d77b0f0c47809e6408b2771adb73 (table rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]) successfully deleted
W20240503 14:16:36.252458 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7f43f3a72cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 14:16:36.272447 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f139d77b0f0c47809e6408b2771adb73 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 14:16:36.279407 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7f4402ec68de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
W20240503 14:16:36.296867 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7f4402ededaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f4403009480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f4403041a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240503 14:16:36.319365 4534 consensus_peers.cc:485] T 1d584b7026b64ba1b720ed21d7019740 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7f4403024981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f440302470e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f43f3a7cb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f43f3a8019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f43f3a80019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240503 14:16:36.342180 4534 consensus_peers.cc:485] T 059a4a35aa94412994bb431342ba0753 P c2cf5d2244244a2db5ebc7dc46fd430d -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
@ 0x7f43f3a5b01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43f6de06db start_thread at ??:0
@ 0x7f43efc1e71f clone at ??:0
W20240503 14:16:36.375309 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:36.393565 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:36.421103 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:36.489912 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:36.570267 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:36.586555 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:36.610433 4315 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:36.649047 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:36.702399 11866 ts_tablet_manager.cc:902] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
W20240503 14:16:36.708920 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:36.734002 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:36.760131 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:36.769048 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:36.773854 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:36.776358 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:36.806327 4334 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:36.818723 11866 tablet_copy_client.cc:308] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:36.819702 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet f139d77b0f0c47809e6408b2771adb73 from peer 7f22b4d5b1b5466dbbdd344f0721c05f ({username='slave'} at 127.20.99.1:48499)
I20240503 14:16:36.819898 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet f139d77b0f0c47809e6408b2771adb73 from peer 7f22b4d5b1b5466dbbdd344f0721c05f at {username='slave'} at 127.20.99.1:48499: session id = 7f22b4d5b1b5466dbbdd344f0721c05f-f139d77b0f0c47809e6408b2771adb73
I20240503 14:16:36.822831 2712 tablet_copy_source_session.cc:205] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:36.824433 11866 ts_tablet_manager.cc:1875] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:36.825618 2692 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:36.826452 11455 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:36.830482 11866 ts_tablet_manager.cc:1888] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:16:36.830673 11866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f139d77b0f0c47809e6408b2771adb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:36.836084 4315 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:36.842191 11866 tablet_copy_client.cc:783] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:36.842456 11866 tablet_copy_client.cc:647] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:36.845857 11866 tablet_copy_client.cc:523] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:36.849835 11866 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
W20240503 14:16:36.865085 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:36.887751 2498 connection.cc:535] client connection to 127.20.99.8:45929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:36.888003 4405 connection.cc:535] client connection to 127.20.99.8:45929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:36.888638 2237 connection.cc:535] client connection to 127.20.99.8:45929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:36.888876 2628 connection.cc:535] client connection to 127.20.99.8:45929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:36.890923 4405 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.891012 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:36.891095 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.891448 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:36.891530 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.891700 2368 connection.cc:535] client connection to 127.20.99.8:45929 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:36.894119 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.894212 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:36.894296 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.894354 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.894927 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:36.895046 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:36.899438 2692 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:36.908262 2159 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:16:36.910454 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:36.914714 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:36.885437 11889 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:36.925315 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:36.956735 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:36.975636 2542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:36.998821 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 32927a23842b4725b9dfafbac47fe754 with delete_type TABLET_DATA_TOMBSTONED (TS b35c386497d046de9be5007a31ab7d10 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35800
W20240503 14:16:37.007483 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.017921 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:37.022809 11912 tablet_replica.cc:330] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: stopping tablet replica
I20240503 14:16:37.023056 11912 raft_consensus.cc:2229] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:37.033214 11912 raft_consensus.cc:2258] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:37.034030 11912 ts_tablet_manager.cc:1875] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:37.066308 4334 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
W20240503 14:16:37.068517 4269 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): Couldn't send request to peer 7f22b4d5b1b5466dbbdd344f0721c05f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32927a23842b4725b9dfafbac47fe754. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:37.074520 2692 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:37.075721 11889 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:37.077322 11912 ts_tablet_manager.cc:1888] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:37.077471 11912 log.cc:1196] T 32927a23842b4725b9dfafbac47fe754 P b35c386497d046de9be5007a31ab7d10: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-2/wal/wals/32927a23842b4725b9dfafbac47fe754
I20240503 14:16:37.077926 11889 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:37.078325 2144 catalog_manager.cc:4830] TS b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): tablet 32927a23842b4725b9dfafbac47fe754 (table rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]) successfully deleted
I20240503 14:16:37.081766 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 32927a23842b4725b9dfafbac47fe754 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:37.082592 2692 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:37.087167 2151 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
W20240503 14:16:37.095789 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:37.129701 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:37.190073 11866 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:37.190577 11866 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap complete.
I20240503 14:16:37.191383 11866 ts_tablet_manager.cc:1367] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.342s user 0.097s sys 0.038s
I20240503 14:16:37.192234 11866 raft_consensus.cc:348] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:37.192456 11866 raft_consensus.cc:729] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
W20240503 14:16:37.192731 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.192728 11866 consensus_queue.cc:260] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:37.194479 11866 ts_tablet_manager.cc:1398] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:37.199762 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-f139d77b0f0c47809e6408b2771adb73 received from {username='slave'} at 127.20.99.1:48499
I20240503 14:16:37.199863 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-f139d77b0f0c47809e6408b2771adb73 on tablet f139d77b0f0c47809e6408b2771adb73 with peer 7f22b4d5b1b5466dbbdd344f0721c05f
W20240503 14:16:37.273792 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:37.310695 10288 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.310871 10288 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.311796 10288 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.312253 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
W20240503 14:16:37.313827 4405 leader_election.cc:334] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.313879 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "d4cbc83202c8473d8c897582d30e5989" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:37.314458 4402 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4cbc83202c8473d8c897582d30e5989; no voters: b35c386497d046de9be5007a31ab7d10, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:37.315873 10288 raft_consensus.cc:2726] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:37.343780 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.438336 11493 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.438562 11493 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:37.439687 11493 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:37.443759 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:37.444069 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:37.444165 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:37.444639 11493 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:37.514245 11866 ts_tablet_manager.cc:902] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: Initiating tablet copy from peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:37.547933 11493 raft_consensus.cc:482] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.548107 11493 raft_consensus.cc:504] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.548822 11493 leader_election.cc:288] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.550205 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "285488446ac445c5a5dfe654bfaf9835" is_pre_election: true
I20240503 14:16:37.550614 11866 tablet_copy_client.cc:308] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:42763
I20240503 14:16:37.552150 4355 tablet_copy_service.cc:140] P 285488446ac445c5a5dfe654bfaf9835: Received BeginTabletCopySession request for tablet 32927a23842b4725b9dfafbac47fe754 from peer 7f22b4d5b1b5466dbbdd344f0721c05f ({username='slave'} at 127.20.99.1:33683)
I20240503 14:16:37.552309 4355 tablet_copy_service.cc:161] P 285488446ac445c5a5dfe654bfaf9835: Beginning new tablet copy session on tablet 32927a23842b4725b9dfafbac47fe754 from peer 7f22b4d5b1b5466dbbdd344f0721c05f at {username='slave'} at 127.20.99.1:33683: session id = 7f22b4d5b1b5466dbbdd344f0721c05f-32927a23842b4725b9dfafbac47fe754
W20240503 14:16:37.559455 2498 leader_election.cc:334] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.559608 2498 leader_election.cc:302] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 285488446ac445c5a5dfe654bfaf9835, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:37.560055 11493 raft_consensus.cc:2726] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:37.561302 4355 tablet_copy_source_session.cc:205] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:37.564213 11866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:37.577927 11711 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.578168 11711 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.579380 11711 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.580713 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "d4cbc83202c8473d8c897582d30e5989" is_pre_election: true
I20240503 14:16:37.581033 4468 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
I20240503 14:16:37.581710 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
W20240503 14:16:37.588773 2237 leader_election.cc:334] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.588948 2237 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, d4cbc83202c8473d8c897582d30e5989; no voters: b35c386497d046de9be5007a31ab7d10, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:37.589787 12028 raft_consensus.cc:2726] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:37.592985 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.613243 11866 tablet_copy_client.cc:783] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:37.613624 11866 tablet_copy_client.cc:647] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:37.619863 11866 tablet_copy_client.cc:523] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:37.629837 11866 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
W20240503 14:16:37.630514 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.648836 11711 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.649080 11711 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.650156 11711 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.651701 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
W20240503 14:16:37.655298 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.655449 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, c37183c2965f46a28847c1262eb97829
W20240503 14:16:37.655926 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:37.658229 11711 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:37.658919 2301 raft_consensus.cc:1203] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240503 14:16:37.709509 11866 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:37.709978 11866 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap complete.
I20240503 14:16:37.710260 11866 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.081s user 0.049s sys 0.004s
I20240503 14:16:37.711007 11866 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:37.711237 11866 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:16:37.711516 11866 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:37.712029 11866 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 14:16:37.712677 4355 tablet_copy_service.cc:342] P 285488446ac445c5a5dfe654bfaf9835: Request end of tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-32927a23842b4725b9dfafbac47fe754 received from {username='slave'} at 127.20.99.1:33683
I20240503 14:16:37.712816 4355 tablet_copy_service.cc:433] P 285488446ac445c5a5dfe654bfaf9835: ending tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-32927a23842b4725b9dfafbac47fe754 on tablet 32927a23842b4725b9dfafbac47fe754 with peer 7f22b4d5b1b5466dbbdd344f0721c05f
W20240503 14:16:37.719977 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:37.742465 11493 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.742671 11493 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.743685 11493 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.744743 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:37.745033 2301 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
I20240503 14:16:37.745234 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "d4cbc83202c8473d8c897582d30e5989" is_pre_election: true
I20240503 14:16:37.745445 4468 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
I20240503 14:16:37.746019 2496 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10, d4cbc83202c8473d8c897582d30e5989; no voters:
W20240503 14:16:37.747691 2498 leader_election.cc:334] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.748026 11493 raft_consensus.cc:2781] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:16:37.748085 11493 raft_consensus.cc:482] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.748140 11493 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.750766 11493 raft_consensus.cc:504] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.751894 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f"
I20240503 14:16:37.752106 2301 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.752308 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdff0b719a342cfbde746751bd2a89d" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "d4cbc83202c8473d8c897582d30e5989"
I20240503 14:16:37.752494 4468 raft_consensus.cc:3037] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.752666 11423 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.752813 11423 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:37.753635 11423 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:37.755133 4468 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 3.
I20240503 14:16:37.756314 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "285488446ac445c5a5dfe654bfaf9835" is_pre_election: true
I20240503 14:16:37.757048 2301 raft_consensus.cc:2454] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 3.
I20240503 14:16:37.757143 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:16:37.758678 2628 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c37183c2965f46a28847c1262eb97829; no voters: 285488446ac445c5a5dfe654bfaf9835, 685c6181561b4056abdecfd6d8bdb3b2
I20240503 14:16:37.759002 11423 raft_consensus.cc:2726] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:37.759624 2496 leader_election.cc:302] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, b35c386497d046de9be5007a31ab7d10, d4cbc83202c8473d8c897582d30e5989; no voters:
W20240503 14:16:37.761319 2628 leader_election.cc:334] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.771385 11423 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.771567 11423 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.771746 11493 leader_election.cc:288] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 election: Requested vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.772303 11423 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.772317 11493 raft_consensus.cc:2781] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:37.772425 11493 raft_consensus.cc:686] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 3 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:16:37.772689 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:37.772681 11493 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.772927 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:16:37.774232 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:37.783010 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:37.799564 2498 leader_election.cc:334] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.823294 11995 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.823505 11995 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:37.824534 11995 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:37.825212 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:16:37.825433 2691 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 685c6181561b4056abdecfd6d8bdb3b2 in term 2.
I20240503 14:16:37.828051 11889 raft_consensus.cc:482] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.828223 11889 raft_consensus.cc:504] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.829125 11889 leader_election.cc:288] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.829396 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "685c6181561b4056abdecfd6d8bdb3b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "285488446ac445c5a5dfe654bfaf9835" is_pre_election: true
W20240503 14:16:37.832418 2368 leader_election.cc:334] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:37.833216 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.835072 2368 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters: 285488446ac445c5a5dfe654bfaf9835, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:37.835691 11995 raft_consensus.cc:2726] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:37.840636 11889 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.840780 11889 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:37.841606 11889 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.842204 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2" is_pre_election: true
I20240503 14:16:37.842444 2430 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:37.847893 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10" is_pre_election: true
I20240503 14:16:37.848114 2561 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:37.848603 4268 leader_election.cc:302] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:16:37.848948 11889 raft_consensus.cc:2781] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:16:37.849016 11889 raft_consensus.cc:482] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.849084 11889 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.852272 11889 raft_consensus.cc:504] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:37.855649 11889 leader_election.cc:288] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 election: Requested vote from peers b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.863833 2561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fda4460865854505b5220d7ec3b34c57" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "b35c386497d046de9be5007a31ab7d10"
I20240503 14:16:37.864101 2561 raft_consensus.cc:3037] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.866739 2561 raft_consensus.cc:2454] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 3.
I20240503 14:16:37.870615 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:16:37.870872 2691 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 2.
I20240503 14:16:37.871651 4268 leader_election.cc:302] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, b35c386497d046de9be5007a31ab7d10; no voters:
I20240503 14:16:37.872108 11889 raft_consensus.cc:2781] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:37.884441 2151 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 2 to 3, leader changed from c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 3 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
W20240503 14:16:37.887276 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.887504 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:37.887827 11423 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:37.888443 4268 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:37.895103 12081 raft_consensus.cc:2781] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:16:37.895234 12081 raft_consensus.cc:482] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:37.895326 12081 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.901373 11889 raft_consensus.cc:686] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Becoming Leader. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Running, Role: LEADER
I20240503 14:16:37.901785 11889 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:16:37.903367 4271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 70211.1us
W20240503 14:16:37.907685 4271 leader_election.cc:334] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:37.916196 4271 leader_election.cc:334] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:37.916458 4271 leader_election.cc:334] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:37.922626 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.943334 12081 raft_consensus.cc:504] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:37.944348 12081 leader_election.cc:288] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 election: Requested vote from peers 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457), c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:37.945389 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:37.945597 2691 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.948922 2691 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 3.
W20240503 14:16:37.956375 4271 leader_election.cc:334] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.958623 2159 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: term changed from 2 to 3, leader changed from c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) to 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6). New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:16:37.967255 4650 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:15:51.931247 4654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:51.931736 4654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:51.946092 4654 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:55.217443 4654 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:15:57.102221 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.102619 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.102834 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.102968 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.103108 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.103250 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.103343 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.103466 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.103616 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.103813 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.103928 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.104079 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.104243 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.104439 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.104581 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:15:57.488610 4654 rebalancer_tool.cc:1084] tablet dd294a04366145378e3f5e3d306152e3: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:15:57.488842 4654 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:15:59.483240 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.483541 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.483657 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.483767 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.483925 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.484036 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.484241 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.484397 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.484524 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.484676 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.484773 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.484850 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.484946 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.485028 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.485148 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.485285 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.485417 4654 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:00.045135 4654 rebalancer_tool.cc:1074] tablet fe0eebf47c494dad8edf06197cdf15bc: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:00.144635 4654 rebalancer_tool.cc:1074] tablet 944d2a983d6247498b02a6e7b7daf746: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:00.667097 4654 rebalancer_tool.cc:1074] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:01.420455 4654 rebalancer_tool.cc:1074] tablet abdff0b719a342cfbde746751bd2a89d: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:01.894261 4654 rebalancer_tool.cc:1074] tablet f139d77b0f0c47809e6408b2771adb73: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:02.684299 4654 rebalancer_tool.cc:1074] tablet 32927a23842b4725b9dfafbac47fe754: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:02.763392 4654 rebalancer_tool.cc:1074] tablet 1d584b7026b64ba1b720ed21d7019740: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:03.106027 4654 rebalancer_tool.cc:1074] tablet 832f07a6b12740d08fa67d653b3818bf: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:03.761514 4654 rebalancer_tool.cc:1074] tablet fda4460865854505b5220d7ec3b34c57: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:04.117115 4654 rebalancer_tool.cc:1074] tablet 78edc267e9f44d82b7dbb663628359fe: '685c6181561b4056abdecfd6d8bdb3b2' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:04.441546 4654 rebalancer_tool.cc:1074] tablet b359fb8d9afc4c9c856897a722100304: 'b35c386497d046de9be5007a31ab7d10' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:05.072002 4654 rebalancer_tool.cc:1074] tablet aba6537cdb87400cb8b108b1fbfa6c0f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:05.634758 4654 rebalancer_tool.cc:1074] tablet af99824d1853471b867e4ff1f01e9367: 'c37183c2965f46a28847c1262eb97829' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:06.112345 4654 rebalancer_tool.cc:1074] tablet 6325a400641b4d25b95592abb4af7d2d: '685c6181561b4056abdecfd6d8bdb3b2' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:06.575793 4654 rebalancer_tool.cc:1074] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:07.273434 4654 rebalancer_tool.cc:1074] tablet 5f631c7713d24f4caf7550bb05147615: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:07.758610 4654 rebalancer_tool.cc:1074] tablet 65a196c723884d068b3728e876c87eb2: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:08.468407 4654 rebalancer_tool.cc:1074] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:19.879019 4654 rebalancer_tool.cc:1136] tablet f139d77b0f0c47809e6408b2771adb73: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:19.972611 4654 rebalancer_tool.cc:1136] tablet 1d584b7026b64ba1b720ed21d7019740: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move completed: OK
I20240503 14:16:20.231257 4654 rebalancer_tool.cc:1136] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
W20240503 14:16:28.865232 10458 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:28.309635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.5:41737 (local address 127.0.0.1:42552)
0503 14:16:28.367040 (+ 57405us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:28.367063 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:28.367191 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:28.863559 (+496368us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":57278,"thread_start_us":57810,"threads_started":1}
I20240503 14:16:31.437297 4654 rebalancer_tool.cc:1136] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:31.580958 4654 rebalancer_tool.cc:1136] tablet 78edc267e9f44d82b7dbb663628359fe: '685c6181561b4056abdecfd6d8bdb3b2' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:34.040597 4654 rebalancer_tool.cc:1136] tablet aba6537cdb87400cb8b108b1fbfa6c0f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:34.164783 4654 rebalancer_tool.cc:1136] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:34.377853 4654 rebalancer_tool.cc:1136] tablet abdff0b719a342cfbde746751bd2a89d: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move completed: OK
I20240503 14:16:34.471486 4654 rebalancer_tool.cc:1136] tablet 5f631c7713d24f4caf7550bb05147615: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:34.783099 4654 rebalancer_tool.cc:1074] tablet 470b425f5cef494ab3334fe2b85e1c4e: 'b35c386497d046de9be5007a31ab7d10' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:35.302469 4654 rebalancer_tool.cc:1074] tablet a9a474791f9b4585aa0114162705c96f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:35.922184 4654 rebalancer_tool.cc:1074] tablet 9ae190d0f4d843c990e6fdadcac677ed: '685c6181561b4056abdecfd6d8bdb3b2' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:36.058245 4654 rebalancer_tool.cc:1084] tablet b112f8fb89c7404b87a4c6e3d72fdcfd: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move ignored: Illegal state: tablet b112f8fb89c7404b87a4c6e3d72fdcfd: replica 285488446ac445c5a5dfe654bfaf9835 is already present
I20240503 14:16:36.058395 4654 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:36.906344 11767 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:36.696261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.8:45929 (local address 127.0.0.1:57904)
0503 14:16:36.720750 (+ 24489us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:36.720772 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:36.720903 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:36.906081 (+185178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24368,"thread_start_us":37174,"threads_started":1}
Illegal state: tablet server 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): unacceptable health status UNAVAILABLE
I20240503 14:16:37.987409 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0eebf47c494dad8edf06197cdf15bc" candidate_uuid: "285488446ac445c5a5dfe654bfaf9835" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "685c6181561b4056abdecfd6d8bdb3b2"
I20240503 14:16:37.987702 2430 raft_consensus.cc:3037] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:37.990937 2430 raft_consensus.cc:2454] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285488446ac445c5a5dfe654bfaf9835 in term 3.
I20240503 14:16:38.003805 4269 leader_election.cc:302] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 285488446ac445c5a5dfe654bfaf9835, 685c6181561b4056abdecfd6d8bdb3b2, c37183c2965f46a28847c1262eb97829; no voters: c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:38.003975 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:38.004477 11889 raft_consensus.cc:2781] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:38.005021 11889 raft_consensus.cc:686] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Becoming Leader. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Running, Role: LEADER
I20240503 14:16:38.005394 11889 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:38.008127 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:38.025947 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:38.042250 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:38.045526 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:38.049647 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:38.054176 2159 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: term changed from 2 to 3, leader changed from c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) to 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6). New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:38.055250 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:38.067476 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:38.067492 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:38.106524 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:38.131338 2301 raft_consensus.cc:1203] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:16:38.138110 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:38.167423 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:38.186549 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:38.188694 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:38.189890 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:38.237510 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:38.251273 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:38.320636 2301 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 14:16:38.321259 12056 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:38.337615 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:38.343895 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:38.359308 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:38.378799 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:38.380138 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:38.387042 2430 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 14:16:38.387703 11889 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:16:38.392087 4468 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P d4cbc83202c8473d8c897582d30e5989 [term 3 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240503 14:16:38.392654 12056 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:38.413812 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:38.420774 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:38.423918 2561 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 14:16:38.424659 11889 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:38.475469 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:38.476104 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:38.482026 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:38.518501 2691 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
W20240503 14:16:38.524905 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:38.525724 11889 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:38.539795 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:38.545418 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:38.556288 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:38.585060 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:38.622838 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:38.627189 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:38.660271 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:38.726955 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:38.754822 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:38.766769 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:38.768097 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:38.776456 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:38.779100 2430 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:38.806681 4468 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:38.807397 11995 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:38.821226 12109 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:38.823490 4469 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:38.835492 2159 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 6 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:16:38.850049 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:38.856669 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:38.883132 2670 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf4675882e64f35bc3dc9ee06ae366d with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:38.884667 2430 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:38.887122 4469 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:38.887727 11995 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:38.888248 2366 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): Couldn't send request to peer 7f22b4d5b1b5466dbbdd344f0721c05f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:38.890789 11995 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:38.894881 4469 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:38.897399 12204 tablet_replica.cc:330] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:38.897737 12204 raft_consensus.cc:2229] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:38.898113 12204 raft_consensus.cc:2258] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:38.899235 2145 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8bf4675882e64f35bc3dc9ee06ae366d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:38.899636 12204 ts_tablet_manager.cc:1875] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:38.901114 2159 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 7 to 8, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:38.909514 12204 ts_tablet_manager.cc:1888] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:38.909730 12204 log.cc:1196] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:16:38.910785 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet 8bf4675882e64f35bc3dc9ee06ae366d (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
W20240503 14:16:38.917443 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:16:38.919839 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:38.920306 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:38.993876 4405 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:39.003118 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:39.007217 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:39.009835 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:16:39.011343 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:39.054652 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:39.099938 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:39.106065 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:39.123492 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:39.144757 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:39.214870 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240503 14:16:39.250839 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240503 14:16:39.256875 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:39.283536 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:39.287510 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:39.301673 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:39.316927 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:39.319470 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:39.331458 12350 ts_tablet_manager.cc:902] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:39.340301 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:39.344475 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:39.369048 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:39.379977 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:39.398236 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:39.398463 4405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 26 similar messages]
I20240503 14:16:39.402810 12154 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:39.402981 12154 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:39.403653 12154 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:39.407279 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:39.412869 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:39.428805 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:39.429121 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:39.429198 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:39.429581 12154 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:39.434188 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:39.435326 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:39.446238 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:39.461275 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:39.474243 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:39.487332 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:39.507098 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:39.534891 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:39.549038 12350 tablet_copy_client.cc:308] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:39.549777 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer 7f22b4d5b1b5466dbbdd344f0721c05f ({username='slave'} at 127.20.99.1:55961)
I20240503 14:16:39.549997 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer 7f22b4d5b1b5466dbbdd344f0721c05f at {username='slave'} at 127.20.99.1:55961: session id = 7f22b4d5b1b5466dbbdd344f0721c05f-8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:16:39.554801 2452 tablet_copy_source_session.cc:205] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:39.556109 12350 ts_tablet_manager.cc:1875] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:39.562597 12350 ts_tablet_manager.cc:1888] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.5
I20240503 14:16:39.562870 12350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:39.579032 12350 tablet_copy_client.cc:783] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:39.579308 12350 tablet_copy_client.cc:647] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:39.588109 12350 tablet_copy_client.cc:523] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:39.592382 12350 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:16:39.755231 12350 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:39.755684 12350 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap complete.
I20240503 14:16:39.756139 12350 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.164s user 0.070s sys 0.003s
I20240503 14:16:39.756960 12350 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:39.757164 12350 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:16:39.757373 12350 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:39.758064 12350 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:39.759131 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-8bf4675882e64f35bc3dc9ee06ae366d received from {username='slave'} at 127.20.99.1:55961
I20240503 14:16:39.759294 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-8bf4675882e64f35bc3dc9ee06ae366d on tablet 8bf4675882e64f35bc3dc9ee06ae366d with peer 7f22b4d5b1b5466dbbdd344f0721c05f
W20240503 14:16:39.766613 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:39.767522 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:39.788005 12128 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:39.788192 12128 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:39.789034 12128 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:39.790885 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:16:39.791105 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:16:39.793630 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:39.795235 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:39.814468 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:39.815373 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:39.815634 12128 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:39.836115 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:39.837334 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:39.852196 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:39.859534 2301 raft_consensus.cc:1203] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:16:39.861521 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:39.890837 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:39.909229 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:39.941179 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 57771us
W20240503 14:16:39.944900 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:39.945705 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:40.108798 2692 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8"
dest_uuid: "c37183c2965f46a28847c1262eb97829"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35838
I20240503 14:16:40.108992 2692 raft_consensus.cc:595] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Received request to transfer leadership
W20240503 14:16:40.113399 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:40.117022 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:40.141315 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:40.145032 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:40.146027 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:40.247519 2670 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:40.255613 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:40.274025 2542 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:40.331173 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:40.336344 12513 raft_consensus.cc:979] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: : Instructing follower b35c386497d046de9be5007a31ab7d10 to start an election
I20240503 14:16:40.336598 12513 raft_consensus.cc:1067] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Signalling peer b35c386497d046de9be5007a31ab7d10 to start an election
I20240503 14:16:40.337603 2561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8"
dest_uuid: "b35c386497d046de9be5007a31ab7d10"
from {username='slave'} at 127.20.99.4:45979
I20240503 14:16:40.337785 2561 raft_consensus.cc:482] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:40.337864 2561 raft_consensus.cc:3037] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:40.341404 2561 raft_consensus.cc:504] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:40.342705 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c37183c2965f46a28847c1262eb97829"
I20240503 14:16:40.342901 2692 raft_consensus.cc:3032] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:40.342972 2692 raft_consensus.cc:729] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Running, Role: LEADER
I20240503 14:16:40.343214 2692 consensus_queue.cc:260] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:40.343539 2692 raft_consensus.cc:3037] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:40.346321 2692 raft_consensus.cc:2454] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
I20240503 14:16:40.346902 2495 leader_election.cc:302] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b35c386497d046de9be5007a31ab7d10, c37183c2965f46a28847c1262eb97829; no voters:
I20240503 14:16:40.347255 12154 raft_consensus.cc:2781] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:40.347371 12154 raft_consensus.cc:686] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Becoming Leader. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Running, Role: LEADER
I20240503 14:16:40.347523 2561 leader_election.cc:288] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 2 election: Requested vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:16:40.347643 12154 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:40.349074 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1d30b76c4d43d0a54a49892d09a6f8" candidate_uuid: "b35c386497d046de9be5007a31ab7d10" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "285488446ac445c5a5dfe654bfaf9835"
I20240503 14:16:40.349251 4333 raft_consensus.cc:3037] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:40.351861 4333 raft_consensus.cc:2454] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35c386497d046de9be5007a31ab7d10 in term 2.
I20240503 14:16:40.372710 2159 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 reported cstate change: term changed from 1 to 2, leader changed from c37183c2965f46a28847c1262eb97829 (127.20.99.4) to b35c386497d046de9be5007a31ab7d10 (127.20.99.3). New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:16:40.380185 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:40.385679 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:40.591750 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:40.630884 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:40.662983 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:40.716185 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:40.722260 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:40.731757 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:40.751708 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:40.786077 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:40.794278 2692 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:40.795897 12154 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:40.803637 4333 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:16:40.804378 12427 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:16:40.807071 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:40.824401 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:40.836362 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:40.839905 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:40.858479 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:40.860481 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:40.874235 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:40.880481 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:40.890888 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:40.892189 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:40.902132 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:40.921340 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:40.948704 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:40.953883 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:41.087275 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:41.090808 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:41.113289 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:41.134979 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:41.140650 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:41.166224 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:41.170200 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:41.185922 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.206396 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:41.251719 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.263286 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:41.303712 4405 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:41.337623 12664 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:41.337896 12664 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:41.339002 12664 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:41.341229 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:16:41.341579 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:16:41.348150 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:41.348451 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:41.359892 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:41.360097 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:41.360435 12664 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:41.406059 12154 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:41.406260 12154 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:41.406920 12154 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:41.408798 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:41.408773 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:41.409662 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:41.409756 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:41.409966 12154 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:41.439687 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.441057 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.468784 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:41.477316 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:41.479163 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.572986 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:41.616267 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:41.620015 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:41.642108 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:41.666857 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:41.701247 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.707859 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:41.722926 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:41.744225 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:41.758141 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:41.763108 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:41.772323 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:41.776598 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:41.797816 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:41.804363 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:41.807413 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:41.810022 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:41.826395 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:41.830065 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:41.863914 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:41.870982 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:41.876503 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:41.887255 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:41.898129 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:41.958317 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:42.016539 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:42.046700 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:42.060758 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:42.089407 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:42.098616 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:42.104105 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:42.109906 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:42.137975 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:42.164808 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:42.220887 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:42.241143 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:42.242132 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:42.277145 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:42.303300 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:42.307449 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:42.334970 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:42.335114 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:42.349251 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:42.354931 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:42.412240 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:42.574738 2691 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:16:42.576758 2561 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 14:16:42.577350 12510 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:42.595875 12510 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:42.601133 2562 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:42.612131 2157 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 9 to 10, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:16:42.643710 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet f139d77b0f0c47809e6408b2771adb73 with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:42.644767 2691 consensus_queue.cc:237] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
W20240503 14:16:42.647703 2628 consensus_peers.cc:485] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f139d77b0f0c47809e6408b2771adb73. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:42.648094 2561 raft_consensus.cc:1261] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240503 14:16:42.648626 12821 consensus_queue.cc:1035] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 14:16:42.651571 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:42.652613 12510 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:42.660547 2561 raft_consensus.cc:2932] T f139d77b0f0c47809e6408b2771adb73 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:42.663455 12908 tablet_replica.cc:330] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:42.664216 12908 raft_consensus.cc:2229] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:42.665262 12908 raft_consensus.cc:2258] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:42.666049 12908 ts_tablet_manager.cc:1875] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:42.669749 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f139d77b0f0c47809e6408b2771adb73 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:42.672468 2159 catalog_manager.cc:5484] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 10 to 11, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:42.689045 12908 ts_tablet_manager.cc:1888] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 14:16:42.689229 12908 log.cc:1196] T f139d77b0f0c47809e6408b2771adb73 P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/f139d77b0f0c47809e6408b2771adb73
I20240503 14:16:42.690249 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet f139d77b0f0c47809e6408b2771adb73 (table rebalance_test_table_4 [id=ee7088dfd84c41c0a4bb41017e8218ef]) successfully deleted
W20240503 14:16:42.754943 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:42.763407 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:42.809547 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:42.851902 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:42.931569 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:43.022024 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:16:43.033079 2237 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 53143.2us
I20240503 14:16:43.044824 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:43.049749 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:43.072906 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:43.094352 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:43.103224 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:43.107372 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:43.153792 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:43.168958 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:43.174649 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:43.175248 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:43.199416 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:43.207422 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:43.240798 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:43.243271 13014 ts_tablet_manager.cc:902] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:16:43.258872 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:43.259227 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:43.286736 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:43.295476 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:43.318554 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:43.319198 2670 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:43.319293 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:43.335085 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:43.349190 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:43.371280 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:43.372799 13014 tablet_copy_client.cc:308] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:16:43.373643 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet f139d77b0f0c47809e6408b2771adb73 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:38575)
I20240503 14:16:43.373807 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet f139d77b0f0c47809e6408b2771adb73 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:38575: session id = 285488446ac445c5a5dfe654bfaf9835-f139d77b0f0c47809e6408b2771adb73
I20240503 14:16:43.376593 2712 tablet_copy_source_session.cc:205] T f139d77b0f0c47809e6408b2771adb73 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:43.378711 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:43.383975 13014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f139d77b0f0c47809e6408b2771adb73. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:16:43.392110 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:43.411801 13014 tablet_copy_client.cc:783] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:43.412163 13014 tablet_copy_client.cc:647] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:43.414086 13014 tablet_copy_client.cc:523] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:43.417910 13014 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
W20240503 14:16:43.441212 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:43.490069 13014 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:43.490586 13014 tablet_bootstrap.cc:492] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:16:43.490903 13014 ts_tablet_manager.cc:1367] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.073s user 0.047s sys 0.008s
I20240503 14:16:43.491863 13014 raft_consensus.cc:348] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:43.492471 13014 raft_consensus.cc:729] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:16:43.492770 13014 consensus_queue.cc:260] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:43.496963 13014 ts_tablet_manager.cc:1398] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240503 14:16:43.499821 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:43.501811 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-f139d77b0f0c47809e6408b2771adb73 received from {username='slave'} at 127.20.99.6:38575
I20240503 14:16:43.501911 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-f139d77b0f0c47809e6408b2771adb73 on tablet f139d77b0f0c47809e6408b2771adb73 with peer 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:43.506371 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:43.529325 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:43.564189 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:43.673041 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:43.696774 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:43.714635 2366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 44132.1us
I20240503 14:16:43.742260 12154 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:43.742455 12154 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:43.744191 12154 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:43.746193 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:43.746403 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:43.746503 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:43.746930 12154 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:43.750114 4405 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:43.760033 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:43.779882 4334 raft_consensus.cc:1203] T f139d77b0f0c47809e6408b2771adb73 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240503 14:16:43.807780 12664 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:43.807991 12664 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:43.809022 12664 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:43.809517 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:16:43.810145 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:16:43.812916 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:43.813141 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:43.813252 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:43.813859 12664 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:43.835403 4334 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:43.840595 2301 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:16:43.841214 12724 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:43.860597 13109 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
W20240503 14:16:43.862627 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:43.869995 2301 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:43.874713 2152 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 8 to 9, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:43.890167 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet 32927a23842b4725b9dfafbac47fe754 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:43.890951 13120 tablet_replica.cc:330] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:43.891263 13120 raft_consensus.cc:2229] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:43.891652 13120 raft_consensus.cc:2258] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:43.892722 13120 ts_tablet_manager.cc:1875] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:43.897579 4334 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:43.899580 2301 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240503 14:16:43.900421 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32927a23842b4725b9dfafbac47fe754. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:43.901366 13120 ts_tablet_manager.cc:1888] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 14:16:43.901592 13120 log.cc:1196] T 32927a23842b4725b9dfafbac47fe754 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/32927a23842b4725b9dfafbac47fe754
I20240503 14:16:43.903023 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet 32927a23842b4725b9dfafbac47fe754 (table rebalance_test_table_9 [id=57acf2e591e64cd59ee9b0fb07b0cd78]) successfully deleted
I20240503 14:16:43.905943 13109 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:43.910327 13109 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:16:43.913980 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:43.914255 2301 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:43.918740 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 32927a23842b4725b9dfafbac47fe754 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:43.921277 2152 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 9 to 10, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:16:43.937326 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:16:43.974421 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:44.029881 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:44.067394 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240503 14:16:44.074733 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:44.077509 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:44.077692 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:44.100198 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:44.127743 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:44.143707 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:44.148777 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:44.152419 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:44.152545 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:44.176419 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:44.179363 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:44.192049 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:44.215345 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:44.217736 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:44.218972 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:44.219645 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:44.238857 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:44.247100 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:44.259457 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:44.282476 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:44.295483 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:44.302163 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:44.320852 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240503 14:16:44.330358 13196 ts_tablet_manager.cc:902] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
W20240503 14:16:44.334535 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:44.339766 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:44.402684 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 14:16:44.408620 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:44.408986 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:44.438050 13196 tablet_copy_client.cc:308] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:42763
I20240503 14:16:44.449246 4355 tablet_copy_service.cc:140] P 285488446ac445c5a5dfe654bfaf9835: Received BeginTabletCopySession request for tablet 32927a23842b4725b9dfafbac47fe754 from peer d4cbc83202c8473d8c897582d30e5989 ({username='slave'} at 127.20.99.7:48159)
I20240503 14:16:44.449414 4355 tablet_copy_service.cc:161] P 285488446ac445c5a5dfe654bfaf9835: Beginning new tablet copy session on tablet 32927a23842b4725b9dfafbac47fe754 from peer d4cbc83202c8473d8c897582d30e5989 at {username='slave'} at 127.20.99.7:48159: session id = d4cbc83202c8473d8c897582d30e5989-32927a23842b4725b9dfafbac47fe754
W20240503 14:16:44.451525 4404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:16:44.452358 4404 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:44.453428 4355 tablet_copy_source_session.cc:205] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:16:44.468410 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:44.476116 13196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:44.488591 13196 tablet_copy_client.cc:783] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:44.488826 13196 tablet_copy_client.cc:647] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:44.510177 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:44.522300 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:44.527969 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:44.514340 13196 tablet_copy_client.cc:523] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:44.543658 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:44.556104 13196 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: Bootstrap starting.
I20240503 14:16:44.582279 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:44.602914 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:44.671049 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:44.704694 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:44.707314 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:44.723892 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:44.734767 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:44.736636 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:44.753064 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:44.753976 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:44.778100 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:44.812182 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:44.851985 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:44.888111 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:44.897045 13196 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:44.897462 13196 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: Bootstrap complete.
I20240503 14:16:44.897702 13196 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: Time spent bootstrapping tablet: real 0.342s user 0.053s sys 0.011s
I20240503 14:16:44.898487 13196 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:44.898687 13196 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4cbc83202c8473d8c897582d30e5989, State: Initialized, Role: FOLLOWER
I20240503 14:16:44.898880 13196 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:44.899472 13196 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:16:44.903944 4355 tablet_copy_service.cc:342] P 285488446ac445c5a5dfe654bfaf9835: Request end of tablet copy session d4cbc83202c8473d8c897582d30e5989-32927a23842b4725b9dfafbac47fe754 received from {username='slave'} at 127.20.99.7:48159
I20240503 14:16:44.904148 4355 tablet_copy_service.cc:433] P 285488446ac445c5a5dfe654bfaf9835: ending tablet copy session d4cbc83202c8473d8c897582d30e5989-32927a23842b4725b9dfafbac47fe754 on tablet 32927a23842b4725b9dfafbac47fe754 with peer d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:44.928176 4469 raft_consensus.cc:1203] T 32927a23842b4725b9dfafbac47fe754 P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240503 14:16:44.963974 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:44.993255 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:45.190315 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:45.195603 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:45.258803 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:45.318847 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:45.341146 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:45.383006 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:45.384217 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:45.394773 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:45.400936 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:45.430999 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:45.454211 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:45.462438 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:45.480777 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:45.523384 13265 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:45.523618 13265 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:45.524662 13265 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:16:45.529242 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:45.529482 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:45.529577 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:45.531049 13265 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:45.538018 2412 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:45.538416 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:45.556696 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:45.563635 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:45.563876 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
I20240503 14:16:45.565363 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:45.584617 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:45.615165 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:45.617048 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:45.618892 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:45.625048 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:45.655328 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:45.657363 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 14:16:45.685227 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:45.689705 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:45.706977 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:45.722326 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:45.727041 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:45.731319 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:45.734076 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:45.737049 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:45.752491 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:45.768486 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:45.772831 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:45.836967 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:45.843103 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:45.851442 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:45.906329 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:45.999819 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:46.055032 4405 consensus_peers.cc:485] T 78edc267e9f44d82b7dbb663628359fe P d4cbc83202c8473d8c897582d30e5989 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:46.145059 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:46.190902 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:46.209853 2431 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
W20240503 14:16:46.214226 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:46.227517 4469 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:16:46.230494 13202 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:16:46.246137 13202 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:46.259027 2152 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 8 to 9, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:46.263037 4469 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:46.277287 2562 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
W20240503 14:16:46.279507 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:46.287858 4334 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:16:46.288532 13480 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:16:46.294112 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf4675882e64f35bc3dc9ee06ae366d with delete_type TABLET_DATA_TOMBSTONED (TS 7f22b4d5b1b5466dbbdd344f0721c05f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57042
I20240503 14:16:46.295030 2431 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:46.296757 13497 tablet_replica.cc:330] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: stopping tablet replica
I20240503 14:16:46.296952 13497 raft_consensus.cc:2229] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:46.297119 13497 raft_consensus.cc:2258] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 14:16:46.297583 2366 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): Couldn't send request to peer 7f22b4d5b1b5466dbbdd344f0721c05f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:46.297655 4469 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:16:46.297842 13497 ts_tablet_manager.cc:1875] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:46.298209 13344 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:16:46.298882 13393 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:46.299772 4334 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:46.305424 2156 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 5 to 7, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) evicted. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:16:46.307809 13344 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
W20240503 14:16:46.308749 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:46.310659 13497 ts_tablet_manager.cc:1888] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240503 14:16:46.310760 13497 log.cc:1196] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-0/wal/wals/8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:16:46.311117 2671 tablet_service.cc:1514] Processing DeleteTablet for tablet eb1d30b76c4d43d0a54a49892d09a6f8 with delete_type TABLET_DATA_TOMBSTONED (TS c37183c2965f46a28847c1262eb97829 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60008
I20240503 14:16:46.311498 2145 catalog_manager.cc:4830] TS 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): tablet 8bf4675882e64f35bc3dc9ee06ae366d (table rebalance_test_table_2 [id=c09112e7dcac45ce97ea2b46e7422d89]) successfully deleted
I20240503 14:16:46.312464 4469 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:46.321059 2562 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:16:46.325084 4334 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:16:46.326164 13480 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:16:46.329173 2495 consensus_peers.cc:485] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb1d30b76c4d43d0a54a49892d09a6f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:46.335160 4334 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:46.337023 13500 tablet_replica.cc:330] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: stopping tablet replica
I20240503 14:16:46.337486 13500 raft_consensus.cc:2229] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:16:46.339030 13393 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:46.339504 13500 raft_consensus.cc:2258] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:16:46.341634 13500 ts_tablet_manager.cc:1875] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:16:46.342957 2145 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8bf4675882e64f35bc3dc9ee06ae366d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:46.345059 2156 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P d4cbc83202c8473d8c897582d30e5989 reported cstate change: config changed from index 9 to 10, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:16:46.348732 2151 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 7 to 8, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) added. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:16:46.350922 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet eb1d30b76c4d43d0a54a49892d09a6f8 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:16:46.361305 13500 ts_tablet_manager.cc:1888] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:16:46.361858 13500 log.cc:1196] T eb1d30b76c4d43d0a54a49892d09a6f8 P c37183c2965f46a28847c1262eb97829: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-3/wal/wals/eb1d30b76c4d43d0a54a49892d09a6f8
I20240503 14:16:46.362860 2144 catalog_manager.cc:4830] TS c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): tablet eb1d30b76c4d43d0a54a49892d09a6f8 (table rebalance_test_table_1 [id=273428c6d0c246d587cf9febaf47d1b7]) successfully deleted
W20240503 14:16:46.388080 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:46.414057 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:46.503299 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:46.528470 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:46.528867 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:46.565861 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:46.616560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:46.616938 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:46.650436 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:46.659322 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:46.667896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:46.678601 4449 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:16:46.682785 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:46.700595 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:46.750217 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:46.798586 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:46.815711 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:46.836004 13603 ts_tablet_manager.cc:902] T eb1d30b76c4d43d0a54a49892d09a6f8 P d4cbc83202c8473d8c897582d30e5989: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
F20240503 14:16:46.836393 13603 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-32927a23842b4725b9dfafbac47fe754.queue_time_us, tablet-download-pool-32927a23842b4725b9dfafbac47fe754.queue_time_us) (tablet-download-pool-32927a23842b4725b9dfafbac47fe754.run_cpu_time_us, tablet-download-pool-32927a23842b4725b9dfafbac47fe754.run_cpu_time_us) (tablet-download-pool-32927a23842b4725b9dfafbac47fe754.run_wall_time_us, tablet-download-pool-32927a23842b4725b9dfafbac47fe754.run_wall_time_us) (tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.queue_time_us, tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.queue_time_us) (tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_cpu_time_us, tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_cpu_time_us) (tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_wall_time_us, tablet-download-pool-470b425f5cef494ab3334fe2b85e1c4e.run_wall_time_us) (tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.queue_time_us, tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.queue_time_us) (tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_cpu_time_us, tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_cpu_time_us) (tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_wall_time_us, tablet-download-pool-57c4a133b5c646f78b07f37dc85904f6.run_wall_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.queue_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.queue_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_cpu_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_cpu_time_us) (tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_wall_time_us, tablet-download-pool-65a196c723884d068b3728e876c87eb2.run_wall_time_us) (tablet-download-pool-75bafc7675ca45639b696db6e4b7ce36.queue_time_us, tablet-download-pool-75bafc7675ca45639b696db6e4b7ce36.queue_time_us) (tablet-download-pool-75bafc7675ca45639b696db6e4b7ce36.run_cpu_time_us, tablet-download-pool-75bafc7675ca45639b696db6e4b7ce36.run_cpu_time_us) (tablet-download-pool-75bafc7675ca45639b696db6e4b7ce36.run_wall_time_us, tablet-download-pool-75bafc7675ca45639b696db6e4b7ce36.run_wall_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.queue_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.queue_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_cpu_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_cpu_time_us) (tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_wall_time_us, tablet-download-pool-78edc267e9f44d82b7dbb663628359fe.run_wall_time_us) (tablet-download-pool-8bf4675882e64f35bc3dc9ee06ae366d.queue_time_us, tablet-download-pool-8bf4675882e64f35bc3dc9ee06ae366d.queue_time_us) (tablet-download-pool-8bf4675882e64f35bc3dc9ee06ae366d.run_cpu_time_us, tablet-download-pool-8bf4675882e64f35bc3dc9ee06ae366d.run_cpu_time_us) (tablet-download-pool-8bf4675882e64f35bc3dc9ee06ae366d.run_wall_time_us, tablet-download-pool-8bf4675882e64f35bc3dc9ee06ae366d.run_wall_time_us) (tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.queue_time_us, tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.queue_time_us) (tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_cpu_time_us, tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_cpu_time_us) (tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_wall_time_us, tablet-download-pool-aba6537cdb87400cb8b108b1fbfa6c0f.run_wall_time_us) (tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.queue_time_us, tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.queue_time_us) (tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_cpu_time_us, tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_cpu_time_us) (tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_wall_time_us, tablet-download-pool-abdff0b719a342cfbde746751bd2a89d.run_wall_time_us) (tablet-download-pool-af99824d1853471b867e4ff1f01e9367.queue_time_us, tablet-download-pool-af99824d1853471b867e4ff1f01e9367.queue_time_us) (tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_cpu_time_us, tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_cpu_time_us) (tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_wall_time_us, tablet-download-pool-af99824d1853471b867e4ff1f01e9367.run_wall_time_us) (tablet-download-pool-b359fb8d9afc4c9c856897a722100304.queue_time_us, tablet-download-pool-b359fb8d9afc4c9c856897a722100304.queue_time_us) (tablet-download-pool-b359fb8d9afc4c9c856897a722100304.run_cpu_time_us, tablet-download-pool-b359fb8d9afc4c9c856897a722100304.run_cpu_time_us) (tablet-download-pool-b359fb8d9afc4c9c856897a722100304.run_wall_time_us, tablet-download-pool-b359fb8d9afc4c9c856897a722100304.run_wall_time_us) (tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.queue_time_us, tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.queue_time_us) (tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_cpu_time_us, tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_cpu_time_us) (tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_wall_time_us, tablet-download-pool-dc88c5dc94844f1c8400c31ad1165a32.run_wall_time_us) (tablet-download-pool-dd294a04366145378e3f5e3d306152e3.queue_time_us, tablet-download-pool-dd294a04366145378e3f5e3d306152e3.queue_time_us) (tablet-download-pool-dd294a04366145378e3f5e3d306152e3.run_cpu_time_us, tablet-download-pool-dd294a04366145378e3f5e3d306152e3.run_cpu_time_us) (tablet-download-pool-dd294a04366145378e3f5e3d306152e3.run_wall_time_us, tablet-download-pool-dd294a04366145378e3f5e3d306152e3.run_wall_time_us) (tablet-download-pool-eb1d30b76c4d43d0a54a49892d09a6f8.queue_time_us, tablet-download-pool-eb1d30b76c4d43d0a54a49892d09a6f8.queue_time_us) (tablet-download-pool-f139d77b0f0c47809e6408b2771adb73.queue_time_us, tablet-download-pool-f139d77b0f0c47809e6408b2771adb73.queue_time_us) (tablet-download-pool-f139d77b0f0c47809e6408b2771adb73.run_cpu_time_us, tablet-download-pool-f139d77b0f0c47809e6408b2771adb73.run_cpu_time_us) (tablet-download-pool-f139d77b0f0c47809e6408b2771adb73.run_wall_time_us, tablet-download-pool-f139d77b0f0c47809e6408b2771adb73.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f3cd9125dcd google::LogMessage::Fail() at ??:0
W20240503 14:16:46.852139 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
@ 0x7f3cd9129b93 google::LogMessage::SendToLog() at ??:0
W20240503 14:16:46.856408 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
@ 0x7f3cd91257cc google::LogMessage::Flush() at ??:0
@ 0x7f3cd9126f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3cdbe66742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f3cdbe32f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f3cdbe2ccd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f3ceb2808de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f3ceb298daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
W20240503 14:16:46.891117 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
@ 0x7f3ceb3c3480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240503 14:16:46.897058 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
@ 0x7f3ceb3fba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3ceb3de981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f3ceb3de70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f3cdbe36b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3cdbe3a19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20240503 14:16:46.927170 4404 consensus_peers.cc:462] T dd294a04366145378e3f5e3d306152e3 P d4cbc83202c8473d8c897582d30e5989 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:46.944885 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:46.957182 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
@ 0x7f3cdbe3a019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f3cdbe1501e kudu::Thread::SuperviseThread() at ??:0
I20240503 14:16:46.976182 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:46.980010 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7f3cdf19a6db start_thread at ??:0
@ 0x7f3cd7fd871f clone at ??:0
W20240503 14:16:46.999747 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:47.081738 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:47.084594 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:47.091042 13368 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:47.091269 13368 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:47.092444 13368 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:47.093452 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:16:47.093734 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:16:47.097564 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:47.097890 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:47.098024 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:47.098656 13368 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:47.101127 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:47.216881 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:47.243670 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:47.266722 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:47.271744 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:47.300572 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:47.306056 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:47.332377 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:47.333745 13665 ts_tablet_manager.cc:902] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
I20240503 14:16:47.340843 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:47.429001 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:47.444375 13665 tablet_copy_client.cc:308] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:16:47.445240 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer 7f22b4d5b1b5466dbbdd344f0721c05f ({username='slave'} at 127.20.99.1:55961)
I20240503 14:16:47.445389 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer 7f22b4d5b1b5466dbbdd344f0721c05f at {username='slave'} at 127.20.99.1:55961: session id = 7f22b4d5b1b5466dbbdd344f0721c05f-8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:16:47.448279 2452 tablet_copy_source_session.cc:205] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:16:47.449867 13665 ts_tablet_manager.cc:1875] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:16:47.457671 13665 ts_tablet_manager.cc:1888] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240503 14:16:47.457921 13665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:47.460893 13665 tablet_copy_client.cc:783] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 0 data blocks...
I20240503 14:16:47.461067 13665 tablet_copy_client.cc:647] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:16:47.467939 13665 tablet_copy_client.cc:523] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:16:47.475520 13665 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
W20240503 14:16:47.490113 2495 connection.cc:535] client connection to 127.20.99.7:44387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:47.490607 2365 connection.cc:535] client connection to 127.20.99.7:44387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:47.493806 4268 connection.cc:535] client connection to 127.20.99.7:44387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:47.494292 2234 connection.cc:535] client connection to 127.20.99.7:44387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:47.528318 12637 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:47.491342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.7:44387 (local address 127.20.99.3:41475)
0503 14:16:47.491625 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:47.491643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:47.491759 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:47.527487 (+ 35728us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":168}
W20240503 14:16:47.529155 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.529266 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.529330 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:16:47.531499 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:47.492069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.7:44387 (local address 127.20.99.2:33703)
0503 14:16:47.492350 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:47.492365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:47.492464 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:47.530678 (+ 38214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":189}
W20240503 14:16:47.531693 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.531771 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.531826 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.531878 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.531939 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.532900 13636 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:47.494968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.7:44387 (local address 127.20.99.1:39195)
0503 14:16:47.495166 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:47.495197 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:47.495313 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:47.532292 (+ 36979us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":108}
W20240503 14:16:47.534189 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.534885 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:47.535938 12816 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:47.495615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.7:44387 (local address 127.20.99.6:42723)
0503 14:16:47.495907 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:47.495923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:47.496026 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:47.535262 (+ 39236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":199}
W20240503 14:16:47.536262 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.536417 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:16:47.566402 13665 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:16:47.566936 13665 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap complete.
I20240503 14:16:47.567301 13665 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.092s user 0.058s sys 0.020s
I20240503 14:16:47.568222 13665 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:47.568516 13665 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:16:47.568789 13665 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:16:47.570789 13665 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:16:47.571792 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-8bf4675882e64f35bc3dc9ee06ae366d received from {username='slave'} at 127.20.99.1:55961
I20240503 14:16:47.571956 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-8bf4675882e64f35bc3dc9ee06ae366d on tablet 8bf4675882e64f35bc3dc9ee06ae366d with peer 7f22b4d5b1b5466dbbdd344f0721c05f
W20240503 14:16:47.577569 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:47.580293 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:16:47.645942 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:47.649057 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:47.740913 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:47.754695 13621 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:47.754881 13621 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:47.755574 13621 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:47.757325 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:47.761430 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:47.761535 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:47.762007 13621 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:47.762153 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:47.796476 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:47.800639 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:47.802340 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:47.805491 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:47.832147 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:47.875938 2301 raft_consensus.cc:1203] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 14:16:47.910851 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:47.935544 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:47.966631 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:47.971366 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:47.973326 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:47.989629 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:16:48.005906 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:48.047566 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:48.063902 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:48.146816 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240503 14:16:48.158900 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:48.187081 13621 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:48.187328 13621 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:48.188288 13621 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:48.192060 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:48.204741 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:48.204928 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:48.205222 13621 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:48.284812 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:48.302726 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:48.346757 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:48.352936 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:48.381763 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:48.396488 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:48.404490 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:16:48.409996 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:48.411022 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:48.418967 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:48.430553 13368 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:48.430721 13368 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:48.431481 13368 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:48.435129 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:48.436726 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:48.438174 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:48.438275 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:48.438675 13368 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:48.448463 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:48.448836 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:48.460964 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:48.465885 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:48.475998 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:48.558570 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:48.603410 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:48.685732 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:48.719403 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:48.740299 13860 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:48.740574 13860 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:48.741690 13860 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:48.742661 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:48.742959 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:16:48.745671 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:48.745900 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:48.746009 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:48.746531 13860 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:48.762874 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:48.783748 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:48.785925 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:48.799567 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:48.808239 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:48.823629 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:48.855549 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:48.890064 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:48.920039 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:48.926465 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:48.942174 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:48.964543 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:49.014940 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:49.021667 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:49.033200 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:49.039744 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 14:16:49.050089 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:49.070284 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:49.072285 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:49.116514 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:49.125979 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:49.130102 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 83 similar messages]
W20240503 14:16:49.181339 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:49.197207 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:49.218621 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:49.306033 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:49.337422 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:49.376272 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:49.406415 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:49.417765 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:49.427253 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:16:49.445140 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:49.446002 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:49.478055 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:49.483494 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:49.510879 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:49.516173 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:49.535936 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 14:16:49.553470 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:49.562387 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:49.569180 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:49.606977 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:49.612370 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:49.637738 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:49.648054 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:49.653612 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:49.657605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:49.658957 4647 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:15:51.732909 4655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:51.733388 4655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:51.758332 4655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:55.040549 4655 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:15:57.135576 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.136314 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.138293 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.138500 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.138641 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.139027 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.139338 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.142102 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.142243 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.142390 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.142500 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.142714 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.142832 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.142985 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.143110 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:15:57.627923 4655 rebalancer_tool.cc:1074] tablet 8bf4675882e64f35bc3dc9ee06ae366d: 'c37183c2965f46a28847c1262eb97829' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:15:58.029337 4655 rebalancer_tool.cc:1074] tablet 9974d80e8d31417788e10140de97e722: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:15:58.821429 4655 rebalancer_tool.cc:1074] tablet 6c94404d8d4f45d59958ae6feb36fc24: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:15:59.594097 4655 rebalancer_tool.cc:1074] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:00.039496 4655 rebalancer_tool.cc:1074] tablet abdff0b719a342cfbde746751bd2a89d: 'c37183c2965f46a28847c1262eb97829' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:00.905380 4655 rebalancer_tool.cc:1074] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:01.554996 4655 rebalancer_tool.cc:1074] tablet aba6537cdb87400cb8b108b1fbfa6c0f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:02.015736 4655 rebalancer_tool.cc:1074] tablet 78edc267e9f44d82b7dbb663628359fe: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:02.591922 4655 rebalancer_tool.cc:1084] tablet 32927a23842b4725b9dfafbac47fe754: 'c37183c2965f46a28847c1262eb97829' -> 'd4cbc83202c8473d8c897582d30e5989' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:16:02.592080 4655 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:04.465744 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:04.466058 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:04.466205 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:04.466325 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:04.466447 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:04.467361 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:04.467609 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:04.467742 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:04.467870 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:04.468040 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:04.605543 4655 rebalancer_tool.cc:1074] tablet 944d2a983d6247498b02a6e7b7daf746: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:04.751139 4655 rebalancer_tool.cc:1074] tablet 1d584b7026b64ba1b720ed21d7019740: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:05.099699 4655 rebalancer_tool.cc:1074] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:05.492681 4655 rebalancer_tool.cc:1074] tablet b359fb8d9afc4c9c856897a722100304: '685c6181561b4056abdecfd6d8bdb3b2' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:05.947553 4655 rebalancer_tool.cc:1074] tablet af99824d1853471b867e4ff1f01e9367: 'c37183c2965f46a28847c1262eb97829' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:06.398420 4655 rebalancer_tool.cc:1074] tablet a9a474791f9b4585aa0114162705c96f: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:06.766134 4655 rebalancer_tool.cc:1074] tablet f139d77b0f0c47809e6408b2771adb73: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:06.973824 4655 rebalancer_tool.cc:1074] tablet 32927a23842b4725b9dfafbac47fe754: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:07.217691 4655 rebalancer_tool.cc:1084] tablet 059a4a35aa94412994bb431342ba0753: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '7146de39e348442da4a363e95fbe0789' move ignored: Illegal state: tablet 059a4a35aa94412994bb431342ba0753: replica 7146de39e348442da4a363e95fbe0789 is already present
I20240503 14:16:07.217849 4655 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:08.847051 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:08.847597 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server d4cbc83202c8473d8c897582d30e5989 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:08.847827 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:08.847944 4655 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
I20240503 14:16:24.447005 4655 rebalancer_tool.cc:1136] tablet 9974d80e8d31417788e10140de97e722: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move completed: OK
I20240503 14:16:27.059481 4655 rebalancer_tool.cc:1136] tablet af99824d1853471b867e4ff1f01e9367: 'c37183c2965f46a28847c1262eb97829' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:27.088447 4655 rebalancer_tool.cc:1136] tablet 944d2a983d6247498b02a6e7b7daf746: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move completed: OK
W20240503 14:16:28.861429 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:27.887506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.5:41737 (local address 127.0.0.1:42548)
0503 14:16:27.940848 (+ 53342us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:27.940869 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:27.941073 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:28.859806 (+918733us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":53190,"thread_start_us":56149,"threads_started":1}
W20240503 14:16:28.915668 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:28.862205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.5:41737 (local address 127.0.0.1:42576)
0503 14:16:28.862518 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:28.862539 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:28.862655 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:28.915466 (+ 52811us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":172}
I20240503 14:16:33.167160 4655 rebalancer_tool.cc:1136] tablet abdff0b719a342cfbde746751bd2a89d: 'c37183c2965f46a28847c1262eb97829' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:37.154009 4655 rebalancer_tool.cc:1136] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:38.940940 4655 rebalancer_tool.cc:1136] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:38.965814 4655 rebalancer_tool.cc:1074] tablet eb1d30b76c4d43d0a54a49892d09a6f8: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
I20240503 14:16:39.016016 4655 rebalancer_tool.cc:1074] tablet 832f07a6b12740d08fa67d653b3818bf: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:39.029425 4655 rebalancer_tool.cc:1136] tablet 832f07a6b12740d08fa67d653b3818bf: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:41.170946 4655 rebalancer_tool.cc:1136] tablet b359fb8d9afc4c9c856897a722100304: '685c6181561b4056abdecfd6d8bdb3b2' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
W20240503 14:16:41.871292 12683 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.20.99.7:44387, user_credentials={real_user=slave}} blocked reactor thread for 49786.1us
I20240503 14:16:42.712674 4655 rebalancer_tool.cc:1123] tablet 1d584b7026b64ba1b720ed21d7019740: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:42.819928 4655 rebalancer_tool.cc:1136] tablet 78edc267e9f44d82b7dbb663628359fe: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:42.935134 4655 rebalancer_tool.cc:1136] tablet aba6537cdb87400cb8b108b1fbfa6c0f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move completed: OK
I20240503 14:16:43.148820 4655 rebalancer_tool.cc:1136] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move completed: OK
I20240503 14:16:43.391651 4655 rebalancer_tool.cc:1136] tablet 6c94404d8d4f45d59958ae6feb36fc24: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move completed: OK
I20240503 14:16:43.479369 4655 rebalancer_tool.cc:1136] tablet a9a474791f9b4585aa0114162705c96f: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move completed: OK
I20240503 14:16:43.534492 4655 rebalancer_tool.cc:1136] tablet 32927a23842b4725b9dfafbac47fe754: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:43.720867 4655 rebalancer_tool.cc:1136] tablet 8bf4675882e64f35bc3dc9ee06ae366d: 'c37183c2965f46a28847c1262eb97829' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:43.762841 4655 rebalancer_tool.cc:1074] tablet 470b425f5cef494ab3334fe2b85e1c4e: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:43.786645 4655 rebalancer_tool.cc:1074] tablet fe0eebf47c494dad8edf06197cdf15bc: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
W20240503 14:16:47.499881 13655 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:47.382619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.7:44387 (local address 127.0.0.1:51928)
0503 14:16:47.383002 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:47.383021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:47.383435 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:47.499690 (+116255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114}
I20240503 14:16:47.695474 4655 rebalancer_tool.cc:1136] tablet f139d77b0f0c47809e6408b2771adb73: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move completed: OK
I20240503 14:16:47.855257 4655 rebalancer_tool.cc:1084] tablet 059a4a35aa94412994bb431342ba0753: 'b35c386497d046de9be5007a31ab7d10' -> '' move ignored: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:47.855415 4655 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): unacceptable health status UNAVAILABLE
I20240503 14:16:49.671367 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:49.671407 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:49.708853 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:49.709365 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:49.713900 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:49.731166 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:49.827306 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:49.834240 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:49.867573 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:49.881228 13621 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:49.881510 13621 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:49.882403 13621 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:49.885458 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:49.885880 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:49.886018 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:49.886894 13621 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:49.891534 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:49.900327 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:49.931031 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:49.935963 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:49.977577 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:16:49.978263 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:49.995208 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:50.009539 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:50.017472 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:50.071342 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:50.072870 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:50.110306 13621 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:50.110505 13621 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:50.111483 13621 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:50.116317 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:50.125512 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:50.125705 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:50.126052 13621 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:50.145228 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:50.145256 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:16:50.230856 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:50.296811 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:50.323170 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:50.325495 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:50.330852 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:50.361001 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:50.386220 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:50.388684 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:50.412479 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:50.414603 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:50.481014 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:50.491851 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:50.492606 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:50.500777 13368 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:50.500977 13368 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:50.501852 13368 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:16:50.510200 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:50.510785 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:50.510921 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:50.511464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:50.512617 13368 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:50.535429 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:50.537014 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:50.580279 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:50.664798 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:50.721638 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:50.743155 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:50.761878 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:50.768486 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:50.771046 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:50.810809 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:50.814651 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:50.825608 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:50.838969 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:50.859258 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:50.915426 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:50.988816 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:51.034322 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:51.054338 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:51.056885 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:51.071460 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:51.073331 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:51.084290 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:51.100986 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:51.135972 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:51.231053 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:51.250310 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:51.251008 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:51.269006 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:51.272164 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:51.294644 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:51.337415 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:51.337802 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:51.355038 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:51.361095 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:51.362270 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:51.384382 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:51.464890 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:51.501153 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:51.522080 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:51.526230 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:51.559670 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:51.570336 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:51.580587 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:51.586455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:51.589169 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:51.615137 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:51.634586 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:51.648475 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:51.735805 13621 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:51.736008 13621 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:51.736778 13621 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:51.738487 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:51.738981 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:51.739084 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:51.739359 13621 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:51.750941 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:51.759824 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:51.778206 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:51.854290 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:51.866879 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:51.881124 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:51.887506 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:51.934737 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:51.938676 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:51.939801 4651 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:15:52.238087 4656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:52.242097 4656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:52.287549 4656 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:55.462810 4656 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:15:57.220530 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.220733 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.220826 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.221107 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.221241 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.221356 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.221485 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.221585 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.221665 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.221768 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:57.221868 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:57.222008 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.222098 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:57.222245 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:57.222327 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
I20240503 14:15:57.486074 4656 rebalancer_tool.cc:1084] tablet dd294a04366145378e3f5e3d306152e3: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '285488446ac445c5a5dfe654bfaf9835' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:15:57.486271 4656 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:15:59.394934 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.395613 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.395962 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.396075 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.396582 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.396792 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.396929 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.397039 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.397171 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.397387 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.397539 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:59.397640 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.397747 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.397862 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.397966 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:59.398094 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:59.398295 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:59.398420 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:15:59.504628 4656 rebalancer_tool.cc:1074] tablet 944d2a983d6247498b02a6e7b7daf746: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:15:59.833436 4656 rebalancer_tool.cc:1074] tablet b112f8fb89c7404b87a4c6e3d72fdcfd: 'c37183c2965f46a28847c1262eb97829' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:00.136358 4656 rebalancer_tool.cc:1084] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move ignored: Illegal state: tablet 57c4a133b5c646f78b07f37dc85904f6: replica 7146de39e348442da4a363e95fbe0789 is already present
I20240503 14:16:00.136497 4656 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:01.884522 6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 55713us
W20240503 14:16:02.243674 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.244777 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:02.244948 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.245090 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.245306 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:02.245472 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:02.245615 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:02.247375 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:02.247531 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:02.247644 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:02.247833 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.247951 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:02.248085 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.248191 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:02.248317 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:02.832957 4656 rebalancer_tool.cc:1074] tablet 65a196c723884d068b3728e876c87eb2: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:03.244895 4656 rebalancer_tool.cc:1074] tablet 6325a400641b4d25b95592abb4af7d2d: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:16:03.394980 4656 rebalancer_tool.cc:1084] tablet 832f07a6b12740d08fa67d653b3818bf: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move ignored: Illegal state: tablet 832f07a6b12740d08fa67d653b3818bf: replica c2cf5d2244244a2db5ebc7dc46fd430d is already present
I20240503 14:16:03.395166 4656 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:05.504647 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:05.504817 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:05.504902 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:05.505025 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:05.505249 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:05.505419 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:05.505530 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:05.505642 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:05.879967 4656 rebalancer_tool.cc:1084] tablet 059a4a35aa94412994bb431342ba0753: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:16:05.880218 4656 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:07.525537 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:07.525851 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:07.525944 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:07.526033 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:07.526172 4656 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:07.680285 4656 rebalancer_tool.cc:1074] tablet dd294a04366145378e3f5e3d306152e3: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:07.855634 4656 rebalancer_tool.cc:1074] tablet 9974d80e8d31417788e10140de97e722: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:07.992040 4656 rebalancer_tool.cc:1074] tablet 1d584b7026b64ba1b720ed21d7019740: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
I20240503 14:16:08.208695 4656 rebalancer_tool.cc:1074] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:12.916858 4656 rebalancer_tool.cc:1136] tablet 65a196c723884d068b3728e876c87eb2: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:20.109176 4656 rebalancer_tool.cc:1136] tablet 6325a400641b4d25b95592abb4af7d2d: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
I20240503 14:16:27.249096 4656 rebalancer_tool.cc:1136] tablet 944d2a983d6247498b02a6e7b7daf746: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:27.376515 4656 rebalancer_tool.cc:1136] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:27.611367 4656 rebalancer_tool.cc:1136] tablet b112f8fb89c7404b87a4c6e3d72fdcfd: 'c37183c2965f46a28847c1262eb97829' -> '285488446ac445c5a5dfe654bfaf9835' move completed: OK
I20240503 14:16:27.808952 4656 rebalancer_tool.cc:1136] tablet 1d584b7026b64ba1b720ed21d7019740: 'c37183c2965f46a28847c1262eb97829' -> '' move completed: OK
I20240503 14:16:27.982754 4656 rebalancer_tool.cc:1136] tablet dd294a04366145378e3f5e3d306152e3: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move completed: OK
I20240503 14:16:28.260272 4656 rebalancer_tool.cc:1074] tablet f139d77b0f0c47809e6408b2771adb73: 'd4cbc83202c8473d8c897582d30e5989' -> '' move scheduled
I20240503 14:16:28.654527 4656 rebalancer_tool.cc:1074] tablet 5f631c7713d24f4caf7550bb05147615: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:28.779443 4656 rebalancer_tool.cc:1074] tablet 75bafc7675ca45639b696db6e4b7ce36: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
I20240503 14:16:28.927075 4656 rebalancer_tool.cc:1074] tablet 8bf4675882e64f35bc3dc9ee06ae366d: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:36.021901 4656 rebalancer_tool.cc:1136] tablet 5f631c7713d24f4caf7550bb05147615: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:37.273361 4656 rebalancer_tool.cc:1136] tablet 9974d80e8d31417788e10140de97e722: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move completed: OK
I20240503 14:16:37.330130 4656 rebalancer_tool.cc:1074] tablet ff59573e2c8f472585b96ca0a0fd75b1: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:37.451627 4656 rebalancer_tool.cc:1074] tablet 32927a23842b4725b9dfafbac47fe754: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:38.947515 4656 rebalancer_tool.cc:1136] tablet 32927a23842b4725b9dfafbac47fe754: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move completed: OK
I20240503 14:16:39.832298 4656 rebalancer_tool.cc:1136] tablet 75bafc7675ca45639b696db6e4b7ce36: 'c37183c2965f46a28847c1262eb97829' -> '' move completed: OK
I20240503 14:16:39.902101 4656 rebalancer_tool.cc:1136] tablet f139d77b0f0c47809e6408b2771adb73: 'd4cbc83202c8473d8c897582d30e5989' -> '' move completed: OK
I20240503 14:16:39.954032 4656 rebalancer_tool.cc:1074] tablet fe0eebf47c494dad8edf06197cdf15bc: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:40.013993 4656 rebalancer_tool.cc:1074] tablet 78edc267e9f44d82b7dbb663628359fe: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:40.050891 4656 rebalancer_tool.cc:1074] tablet 832f07a6b12740d08fa67d653b3818bf: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:48.121364 4656 rebalancer_tool.cc:1123] tablet 78edc267e9f44d82b7dbb663628359fe: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:50.698490 4656 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): unacceptable health status UNAVAILABLE
I20240503 14:16:51.941177 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:51.944628 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:51.972438 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:52.038516 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:52.100788 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:52.105937 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:52.117954 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:52.123199 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:52.129794 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:52.141896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:52.145337 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:52.171943 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:52.180320 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:52.197355 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:52.246084 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:52.285451 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:52.300697 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:52.302956 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:52.335390 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:52.347885 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:52.379586 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:52.398484 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:52.403795 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:52.404853 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:52.404983 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:52.439275 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:52.454551 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:52.464318 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:52.465229 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:16:52.465502 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:52.498277 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:52.526418 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:52.561587 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:52.568958 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:52.575573 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:52.581807 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:52.713296 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:52.787125 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:16:52.813567 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:52.821079 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:52.841909 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:52.856922 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:52.870527 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:52.891340 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:52.913072 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:52.913424 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:52.917565 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:52.962558 13368 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:52.962798 13368 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:52.964072 13368 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:16:52.968323 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:52.977804 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:52.990240 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:52.990397 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:52.990674 13368 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:52.994339 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:52.999536 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:53.008417 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:53.015700 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:53.050385 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:16:53.059686 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:53.149391 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:53.183554 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:16:53.239500 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:53.274080 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:16:53.284060 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:53.288242 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:53.295459 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:53.310652 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:53.319096 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:16:53.319696 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:53.325836 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:53.342792 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:53.366916 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:53.377107 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:16:53.414973 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:53.499522 4648 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskHTWOmZ/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240503 14:15:51.673766 4652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:51.674170 4652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:51.689903 4652 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:54.514127 4652 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:15:55.847638 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:55.847996 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:55.848196 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:55.848366 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:55.848536 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:55.848737 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:55.848860 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:55.848984 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:55.849126 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:55.849261 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:15:55.849402 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:15:55.849673 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:15:55.849814 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:15:55.849954 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:15:56.481254 4652 rebalancer_tool.cc:1074] tablet 944d2a983d6247498b02a6e7b7daf746: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:15:57.001325 4652 rebalancer_tool.cc:1074] tablet 1d584b7026b64ba1b720ed21d7019740: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:15:57.417584 4652 rebalancer_tool.cc:1074] tablet dd294a04366145378e3f5e3d306152e3: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:15:57.827843 4652 rebalancer_tool.cc:1074] tablet eb1d30b76c4d43d0a54a49892d09a6f8: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:15:58.432310 4652 rebalancer_tool.cc:1074] tablet ff59573e2c8f472585b96ca0a0fd75b1: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move scheduled
I20240503 14:15:58.922724 4652 rebalancer_tool.cc:1074] tablet fe0eebf47c494dad8edf06197cdf15bc: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:15:59.634882 4652 rebalancer_tool.cc:1074] tablet aba6537cdb87400cb8b108b1fbfa6c0f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move scheduled
I20240503 14:16:00.259872 4652 rebalancer_tool.cc:1074] tablet 9ae190d0f4d843c990e6fdadcac677ed: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move scheduled
I20240503 14:16:00.977603 4652 rebalancer_tool.cc:1084] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:16:00.977790 4652 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:02.816608 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:02.817029 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server d4cbc83202c8473d8c897582d30e5989
W20240503 14:16:02.817150 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:02.817237 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:02.817345 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 37ac6a3a785d4dfead9698313d2651c9: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.817435 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.817512 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:02.817714 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:02.817802 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:02.817901 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 497c07d387f94011ae892d01054b8a2d: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:02.818025 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 39fac270fc164b88879eae47f2f4feb9: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:02.979578 4652 rebalancer_tool.cc:1074] tablet 75bafc7675ca45639b696db6e4b7ce36: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:03.039713 4652 rebalancer_tool.cc:1074] tablet abdff0b719a342cfbde746751bd2a89d: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
I20240503 14:16:03.865906 4652 rebalancer_tool.cc:1074] tablet 5f631c7713d24f4caf7550bb05147615: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:04.030089 4652 rebalancer_tool.cc:1084] tablet 6325a400641b4d25b95592abb4af7d2d: '685c6181561b4056abdecfd6d8bdb3b2' -> '7146de39e348442da4a363e95fbe0789' move ignored: Illegal state: tablet 6325a400641b4d25b95592abb4af7d2d: replica 7146de39e348442da4a363e95fbe0789 is already present
I20240503 14:16:04.030225 4652 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:05.629364 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:05.629909 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:05.630028 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:05.630199 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server b35c386497d046de9be5007a31ab7d10 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:05.630350 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:05.630546 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 273428c6d0c246d587cf9febaf47d1b7: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server 285488446ac445c5a5dfe654bfaf9835
I20240503 14:16:05.757891 4652 rebalancer_tool.cc:1074] tablet f139d77b0f0c47809e6408b2771adb73: 'd4cbc83202c8473d8c897582d30e5989' -> '' move scheduled
I20240503 14:16:05.848718 4652 rebalancer_tool.cc:1074] tablet 32927a23842b4725b9dfafbac47fe754: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:06.255960 4652 rebalancer_tool.cc:1074] tablet 059a4a35aa94412994bb431342ba0753: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move scheduled
I20240503 14:16:06.674399 4652 rebalancer_tool.cc:1084] tablet a9a474791f9b4585aa0114162705c96f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 14:16:06.674530 4652 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 14:16:08.591883 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee7088dfd84c41c0a4bb41017e8218ef: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:08.592730 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92131433a8e24e6498f234341a021065: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server 7146de39e348442da4a363e95fbe0789
W20240503 14:16:08.593446 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c09112e7dcac45ce97ea2b46e7422d89: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server 285488446ac445c5a5dfe654bfaf9835
W20240503 14:16:08.596279 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b0dd533aba93443fb745a864d9511793: could not find any suitable replica to move from server 7f22b4d5b1b5466dbbdd344f0721c05f to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:08.600497 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ee73b380a5364116800e5a1a41243d03: could not find any suitable replica to move from server 685c6181561b4056abdecfd6d8bdb3b2 to server c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:16:08.600781 4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 57acf2e591e64cd59ee9b0fb07b0cd78: could not find any suitable replica to move from server c37183c2965f46a28847c1262eb97829 to server d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:08.745848 4652 rebalancer_tool.cc:1074] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:15.177464 4652 rebalancer_tool.cc:1136] tablet 32927a23842b4725b9dfafbac47fe754: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move completed: OK
I20240503 14:16:29.120548 4652 rebalancer_tool.cc:1136] tablet aba6537cdb87400cb8b108b1fbfa6c0f: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:29.173846 4652 rebalancer_tool.cc:1136] tablet 944d2a983d6247498b02a6e7b7daf746: 'b35c386497d046de9be5007a31ab7d10' -> '7146de39e348442da4a363e95fbe0789' move completed: OK
W20240503 14:16:36.888442 11801 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:36.332761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.8:45929 (local address 127.0.0.1:45690)
0503 14:16:36.351562 (+ 18801us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:36.351587 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:36.351761 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:36.886911 (+535150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18350,"thread_start_us":18241,"threads_started":1}
I20240503 14:16:36.891378 4652 rebalancer_tool.cc:1123] tablet 1d584b7026b64ba1b720ed21d7019740: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:16:36.982126 4652 rebalancer_tool.cc:1136] tablet dd294a04366145378e3f5e3d306152e3: '7f22b4d5b1b5466dbbdd344f0721c05f' -> 'd4cbc83202c8473d8c897582d30e5989' move completed: OK
I20240503 14:16:37.159003 4652 rebalancer_tool.cc:1123] tablet fe0eebf47c494dad8edf06197cdf15bc: 'c37183c2965f46a28847c1262eb97829' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:37.220640 4652 rebalancer_tool.cc:1074] tablet 78edc267e9f44d82b7dbb663628359fe: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move scheduled
I20240503 14:16:38.925566 4652 rebalancer_tool.cc:1123] tablet 059a4a35aa94412994bb431342ba0753: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:38.960031 4652 rebalancer_tool.cc:1136] tablet 57c4a133b5c646f78b07f37dc85904f6: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move completed: OK
I20240503 14:16:38.975414 4652 rebalancer_tool.cc:1136] tablet 5f631c7713d24f4caf7550bb05147615: 'b35c386497d046de9be5007a31ab7d10' -> 'c2cf5d2244244a2db5ebc7dc46fd430d' move completed: OK
I20240503 14:16:39.015036 4652 rebalancer_tool.cc:1136] tablet 75bafc7675ca45639b696db6e4b7ce36: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move completed: OK
I20240503 14:16:39.071833 4652 rebalancer_tool.cc:1136] tablet abdff0b719a342cfbde746751bd2a89d: 'c37183c2965f46a28847c1262eb97829' -> '' move completed: OK
I20240503 14:16:39.169716 4652 rebalancer_tool.cc:1136] tablet 9ae190d0f4d843c990e6fdadcac677ed: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move completed: OK
I20240503 14:16:39.234261 4652 rebalancer_tool.cc:1136] tablet eb1d30b76c4d43d0a54a49892d09a6f8: '685c6181561b4056abdecfd6d8bdb3b2' -> '285488446ac445c5a5dfe654bfaf9835' move completed: OK
I20240503 14:16:40.512601 4652 rebalancer_tool.cc:1136] tablet f139d77b0f0c47809e6408b2771adb73: 'd4cbc83202c8473d8c897582d30e5989' -> '' move completed: OK
I20240503 14:16:40.608603 4652 rebalancer_tool.cc:1074] tablet 6325a400641b4d25b95592abb4af7d2d: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move scheduled
I20240503 14:16:40.765022 4652 rebalancer_tool.cc:1074] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:40.913978 4652 rebalancer_tool.cc:1074] tablet 9974d80e8d31417788e10140de97e722: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
I20240503 14:16:41.065184 4652 rebalancer_tool.cc:1074] tablet 832f07a6b12740d08fa67d653b3818bf: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:41.282546 4652 rebalancer_tool.cc:1136] tablet 832f07a6b12740d08fa67d653b3818bf: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:41.380594 4652 rebalancer_tool.cc:1136] tablet 61b63f87fb384bb2a48044bcad87c282: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
I20240503 14:16:41.422343 4652 rebalancer_tool.cc:1136] tablet 6325a400641b4d25b95592abb4af7d2d: '685c6181561b4056abdecfd6d8bdb3b2' -> '' move completed: OK
I20240503 14:16:46.076850 4652 rebalancer_tool.cc:1074] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> '' move scheduled
I20240503 14:16:46.169467 4652 rebalancer_tool.cc:1074] tablet 8bf4675882e64f35bc3dc9ee06ae366d: 'c37183c2965f46a28847c1262eb97829' -> '' move scheduled
I20240503 14:16:46.231062 4652 rebalancer_tool.cc:1136] tablet 8bf4675882e64f35bc3dc9ee06ae366d: 'c37183c2965f46a28847c1262eb97829' -> '' move completed: OK
I20240503 14:16:46.364598 4652 rebalancer_tool.cc:1136] tablet dc88c5dc94844f1c8400c31ad1165a32: 'b35c386497d046de9be5007a31ab7d10' -> '' move completed: OK
W20240503 14:16:47.498001 13623 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:16:47.076010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.99.7:44387 (local address 127.0.0.1:51916)
0503 14:16:47.076176 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0503 14:16:47.076192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 14:16:47.076279 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:16:47.497745 (+421466us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":89}
I20240503 14:16:50.729321 4652 rebalancer_tool.cc:1123] tablet 78edc267e9f44d82b7dbb663628359fe: '7f22b4d5b1b5466dbbdd344f0721c05f' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:52.580781 4652 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): unacceptable health status UNAVAILABLE
I20240503 14:16:53.514475 14307 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:53.514679 14307 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:16:53.515374 14307 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:16:53.517077 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:53.517433 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:53.517557 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:53.517791 14307 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:53.526682 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:53.547897 13860 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:53.548075 13860 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:53.549042 13860 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:53.551098 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:53.551363 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:16:53.552882 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:53.553121 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:53.553231 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:53.553458 13860 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:53.594990 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:53.598567 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:53.605038 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:53.631914 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:53.634382 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:53.714543 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:53.770860 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:16:53.791352 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:53.801481 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:53.801810 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:53.808823 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:53.817831 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:53.820075 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:53.826329 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:16:53.840622 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:53.841001 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:53.913287 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:54.005939 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:16:54.034320 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:54.042613 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:54.044555 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:54.049870 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:54.052448 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:54.072777 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:16:54.073928 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:54.078215 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:16:54.079643 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:54.113920 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:16:54.166422 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:54.168404 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:54.175359 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:16:54.199370 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:54.235347 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:54.236305 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:54.255300 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:54.271849 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:54.275017 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:54.289063 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:54.299024 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:16:54.313859 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:54.352118 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:54.400736 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:54.406744 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:16:54.431553 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 127 similar messages]
I20240503 14:16:54.472604 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:54.475018 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:54.494242 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:54.499495 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:54.504387 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:54.519078 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:54.519680 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:54.527686 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:54.530162 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:54.549098 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:54.567812 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240503 14:16:54.580339 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:54.595136 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:54.612321 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:54.616144 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:16:54.674644 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:54.704277 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:54.706907 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:54.711558 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:54.719815 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:54.745092 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:54.753998 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:54.771732 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:54.784384 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:54.899420 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:54.940711 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:54.941351 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:54.956319 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:54.958669 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:54.982785 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:54.999750 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:16:55.001065 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:55.011307 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:55.039121 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:55.102767 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:55.144896 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:55.153975 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:55.164408 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:55.167368 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:55.171294 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:55.178493 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:55.194629 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:55.340945 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:55.363592 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:16:55.396036 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:55.427428 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:55.427965 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:55.432513 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:55.442561 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:55.450670 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:16:55.455709 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:55.529654 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:55.553639 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:16:55.618166 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:55.625156 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:55.638517 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:55.641999 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:55.644114 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:55.700947 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:55.710989 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:55.754976 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:16:55.783751 14837 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:16:55.783932 14837 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:55.784737 14837 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:16:55.785218 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:16:55.785475 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:16:55.786202 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:55.786681 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:55.786803 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:16:55.787042 14837 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:16:55.834101 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:55.836638 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:55.858707 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:55.868546 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:55.878057 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:55.899129 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:55.908509 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:55.942602 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:55.946501 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:55.991317 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:16:56.050547 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:56.053860 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:16:56.064181 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:56.066576 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:56.067055 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:56.090965 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:56.094978 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:16:56.147599 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:56.160442 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:56.170481 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:56.283265 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:16:56.297649 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:16:56.357383 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:56.431787 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:56.450120 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:16:56.457218 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:56.465061 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:16:56.475728 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:56.480888 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:16:56.481520 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:56.484094 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:56.493956 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:56.600975 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:56.601938 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:56.661337 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:56.692607 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:56.705991 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:56.709827 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:56.712584 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:56.745957 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:56.756083 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:16:56.768151 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:56.777115 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:56.787451 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:56.816690 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:56.853542 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:16:56.890167 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:16:56.890422 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:56.908767 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:56.911427 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:56.930860 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:56.933089 13368 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:56.933307 13368 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:56.934362 13368 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:16:56.949204 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:56.956000 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:56.964736 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:56.964897 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:56.965692 13368 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:56.965914 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:56.981796 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.022470 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:57.043509 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:57.043838 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:57.117393 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:57.119647 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:57.135429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:57.138056 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:16:57.139838 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:57.147154 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:57.155015 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:57.157968 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.158367 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:57.163340 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.171594 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.195441 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:57.327432 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:57.336601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:57.360200 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:57.364313 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:57.380281 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:57.380642 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.388466 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:57.394160 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:57.444152 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.453943 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:57.459022 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:16:57.467459 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:57.522131 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:57.530122 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:57.536049 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:57.550617 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:57.551126 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:57.557099 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:57.658299 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:57.688688 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:57.730022 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:57.744498 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:57.746804 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:57.748384 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:57.751473 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:57.752199 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:57.930681 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:16:57.951308 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:57.951352 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:57.952417 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:57.962596 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:57.982800 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:57.986804 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:58.002178 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:58.023643 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:58.029315 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:58.058024 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:58.074956 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:58.197875 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:58.331215 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:16:58.390089 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:58.397586 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:58.400079 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:16:58.408397 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:58.408885 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:58.413132 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:58.418586 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:58.464637 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:58.482244 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:58.546114 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:58.569882 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:58.588575 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:58.601258 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:58.603070 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:58.609331 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:58.611037 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:58.614450 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:58.619418 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:58.619628 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:58.632723 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:58.661242 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:58.728298 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:16:58.788813 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:58.800261 13368 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:16:58.800449 13368 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:16:58.801330 13368 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:16:58.803814 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:16:58.804725 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:58.804857 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:16:58.805331 13368 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:16:58.808105 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:58.827158 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:58.832221 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:58.840643 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:58.846546 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:58.859059 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:58.893209 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:59.022812 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.031939 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.038266 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:59.038581 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:16:59.044210 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:59.049582 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:16:59.064280 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:59.064425 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:59.067019 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:59.082891 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:59.108196 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:59.146287 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:16:59.147240 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:59.177836 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:16:59.183637 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:16:59.184464 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:59.222434 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:59.277601 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:16:59.278935 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:59.290895 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:59.295229 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:59.310209 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:59.333765 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:59.350777 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:59.364923 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:59.431720 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:16:59.448726 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:59.454231 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.464426 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:59.487567 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.504083 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:16:59.505481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:59.505568 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:16:59.510023 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:16:59.511114 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:59.514724 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:59.545711 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:59.588560 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:16:59.605939 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:59.612198 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240503 14:16:59.614992 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:59.624994 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:59.646271 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.714874 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:16:59.716476 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:59.719625 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:59.727568 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:16:59.730943 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:16:59.734890 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:16:59.740260 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:16:59.747886 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:16:59.773391 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:16:59.866505 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.883724 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:16:59.902079 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:16:59.922410 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:59.940680 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:16:59.942763 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:59.962852 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:59.963160 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:16:59.987282 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:17:00.042809 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:17:00.116592 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:00.132937 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:17:00.140142 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:00.246392 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:00.266717 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:17:00.328474 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:00.349436 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:00.363250 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:00.369550 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:00.405584 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:00.415403 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:00.460033 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:00.474539 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:00.475270 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:00.490989 15510 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:00.491158 15510 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:00.491818 15510 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:00.493126 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:00.493345 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:00.493454 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:00.493661 15510 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:00.547271 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:00.571235 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:00.583596 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:00.588651 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:00.590636 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:00.601691 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:00.613869 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:00.722319 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:00.723871 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:00.779412 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:00.798760 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:00.805176 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:00.807787 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:00.810766 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:00.815498 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:00.818223 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:00.956745 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:00.998077 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:01.014946 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:01.015411 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:01.017932 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:01.051169 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:01.066128 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:01.071341 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:01.105846 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:01.138346 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:01.144099 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:01.151650 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:01.167131 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:01.236964 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:01.253264 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:01.261947 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:17:01.262658 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:01.266158 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:01.283459 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:01.302709 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:01.306159 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:01.461740 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:01.462031 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:01.482609 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:01.518589 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:01.532816 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:01.563373 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:01.571576 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:01.588472 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:01.596787 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:01.638295 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:01.646711 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:01.661593 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:01.682528 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:01.685755 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:01.711321 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:01.712816 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:01.726001 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:01.745170 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:01.752171 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:01.823889 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:01.836207 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:01.859882 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:01.883219 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:01.904470 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:17:01.920944 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:01.929173 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:01.939087 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:01.940448 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:01.952832 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:01.969048 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:01.972174 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:02.050355 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:02.088680 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:02.132598 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:02.152740 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:02.180932 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:02.207760 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:02.254629 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:02.256698 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:02.264418 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:02.322835 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:02.347293 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:02.361757 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:02.369721 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:02.376876 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:02.423640 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:02.436585 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:02.456805 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:02.511499 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:02.520368 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:02.522132 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:17:02.531371 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:02.541802 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:02.542762 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:17:02.557647 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:02.563439 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:02.584262 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:02.604247 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:02.739782 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:02.754061 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:02.759572 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:02.763274 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:02.764235 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:02.764259 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:02.767592 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:02.783548 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:02.803437 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:17:02.894778 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:02.960419 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:02.974517 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:02.979789 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:02.981451 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:02.994789 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:03.012214 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:03.060362 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:03.070961 15845 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:03.071098 15845 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:03.071689 15845 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:03.072944 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:03.073222 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:03.073315 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:03.073515 15845 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:03.086177 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:03.129101 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:03.133867 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:03.178903 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:03.185678 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:03.189807 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:03.190933 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:03.202762 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:03.217195 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:03.243961 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:03.258457 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:03.288033 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:03.420724 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:03.423559 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:03.424445 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:03.430884 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:03.440379 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:03.488543 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:03.526823 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:17:03.596633 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:03.606842 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:03.638273 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:03.639554 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:03.642292 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:03.665798 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:03.671101 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:03.675891 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:03.710994 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:03.711764 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:03.742830 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:03.768635 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:03.790876 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:03.798776 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:03.877882 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:03.906177 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:03.909400 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:03.911962 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:03.924100 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:03.933017 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:03.940146 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:03.984750 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:04.044061 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:04.054711 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:04.138545 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:04.177103 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.187733 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:17:04.209858 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:17:04.217540 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.225546 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:04.248054 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:04.280473 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:04.327384 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:04.330879 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:04.332440 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.334298 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:04.354101 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:04.378261 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:04.387475 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:04.404419 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:04.405103 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.436357 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.445251 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 14:17:04.512174 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:17:04.548038 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:04.561344 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:04.564644 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:04.564915 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:04.566442 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:04.568758 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:04.582026 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:04.587440 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:04.599833 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:04.632814 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:04.657644 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:17:04.658540 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:04.736442 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:04.751603 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:04.777493 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:04.786881 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:04.793078 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:04.812031 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:04.826071 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:04.833477 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:04.834565 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:04.858567 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.888351 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:04.898609 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:04.988512 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:04.991079 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:04.992311 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:05.003465 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:05.011011 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:17:05.013597 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:05.025800 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:05.080564 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:17:05.119102 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:05.189996 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:05.211889 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:05.216655 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:05.222298 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:05.224892 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:05.227244 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:05.261948 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:17:05.264885 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:05.361186 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:05.409775 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:05.428226 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:05.431421 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:05.457567 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:05.457684 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:05.461012 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:05.467597 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:05.502349 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:05.520321 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:05.621119 16338 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:05.621289 16338 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:05.622092 16338 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:05.623103 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:17:05.623409 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:17:05.624512 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:05.627138 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:05.627278 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:05.627689 16544 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:17:05.655993 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:05.662823 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:05.667696 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:05.671517 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:05.690053 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:05.691619 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:05.700575 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:05.703390 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:05.755100 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:05.763025 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:05.811889 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:05.868126 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:05.882309 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:05.886229 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:05.897040 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:05.897691 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:05.902325 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:06.001391 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:06.071961 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:06.124918 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:06.136574 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:06.186820 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:06.198942 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:06.253552 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:06.287706 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:06.290486 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:06.295579 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:06.303426 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:06.318939 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:06.336437 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:06.370906 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:06.391095 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:06.391121 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:06.398517 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:06.502920 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:06.512430 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:06.532428 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:06.535899 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:06.545632 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:06.546057 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:06.559463 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:06.665479 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:06.685047 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:06.715058 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:06.724735 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:06.734108 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:17:06.737829 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:06.738548 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:06.752957 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:06.753628 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:06.755312 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:06.820125 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:17:06.872109 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:06.887776 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:06.900264 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:17:06.932242 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:17:06.943104 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:06.953826 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:06.962929 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:06.966521 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:06.974910 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:06.975241 16596 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:06.975401 16596 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:06.976151 16596 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:17:06.978170 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:06.991355 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:06.991514 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:06.991762 16754 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:06.992478 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:07.007985 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:07.017969 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:07.021044 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:17:07.021755 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:07.072824 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:07.153237 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:07.165915 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:07.177067 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:07.177913 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:07.196130 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:07.204023 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:07.211563 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:07.224359 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:17:07.234005 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:07.244627 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:07.269802 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:07.300094 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:07.309120 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:17:07.362673 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:17:07.384809 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:07.401757 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:07.402940 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:07.417557 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:07.420864 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:07.426890 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:07.439618 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:07.500017 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:07.578969 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:17:07.602536 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:07.607384 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:07.619472 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:07.640956 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:07.646144 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:17:07.648008 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:07.651324 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:07.712829 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:07.754876 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:17:07.775851 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:07.835767 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:07.838323 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:07.853593 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:07.866959 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:07.871387 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:07.901948 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:07.917874 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:07.934165 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:07.938720 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:07.953732 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:08.023068 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:08.192989 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:08.193177 16239 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:08.193344 16239 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:08.193987 16239 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:08.195358 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.195580 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:08.195663 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:08.195905 16239 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:17:08.260424 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:08.271035 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:08.272215 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:08.293556 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:08.294488 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:08.301818 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.338487 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.355654 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:08.365391 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.402621 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:08.484009 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:08.485560 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:08.485749 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:08.511835 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:08.511914 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:08.533733 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:08.534180 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:08.611271 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:08.715873 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:17:08.725585 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:08.730281 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:08.744931 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:08.747385 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:08.752203 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:08.768816 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.772053 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:08.773466 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:08.809859 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.820914 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:08.858872 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:08.903685 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:08.915653 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:17:08.939977 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:08.940635 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:08.942334 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:08.959035 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:08.959421 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:08.960454 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:08.963961 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:09.045301 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:09.070972 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:09.138412 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:09.141978 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:09.145256 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:09.168448 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:09.197228 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:09.197341 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:09.198686 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:17:09.214625 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240503 14:17:09.219244 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:09.304641 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:09.305945 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:09.322192 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:09.356945 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:17:09.389514 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:09.397557 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:09.408790 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:09.425344 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:09.430369 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:09.443415 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:09.448513 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:17:09.450706 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:09.451517 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:09.510047 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:09.556507 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:09.566679 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:09.590000 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:09.612058 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:09.617831 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:09.625838 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:09.627816 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:09.638898 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:09.656746 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:17:09.660722 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:09.666954 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:09.679138 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:09.679167 17149 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:09.679343 17149 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:09.679915 17149 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:09.682307 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:09.682965 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:09.683055 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:09.683431 17149 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:09.714047 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:09.792764 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:09.817926 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:09.849375 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:09.852963 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:17:09.855603 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:09.871383 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:09.873590 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:09.877653 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:09.878003 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:09.890277 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:09.941112 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:09.943328 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:09.971055 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:10.055296 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:10.103147 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:10.106336 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:10.125289 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:10.149207 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:10.156982 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:10.231899 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:10.270645 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:10.277920 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:10.292418 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:10.302155 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:10.304451 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:10.314442 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:10.354102 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:17:10.368614 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:10.458263 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:10.496292 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:10.500604 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:10.515230 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:10.516207 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:10.540138 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:10.540273 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:10.647346 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:10.728214 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:10.735757 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:10.735991 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:10.753787 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:10.761083 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:10.767479 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:10.769588 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:10.777666 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:10.845039 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:10.916744 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:10.938025 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:10.956766 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:10.969599 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:10.979691 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:10.982196 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:10.983676 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:11.060438 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:11.113230 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:11.128806 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:11.134287 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:11.157456 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:11.161024 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:11.163232 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:11.170549 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:11.175894 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:11.192065 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:11.258705 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:11.284943 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:11.300693 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:11.358557 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:11.395841 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:11.426630 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:11.432088 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:11.448397 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:11.448590 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:11.467952 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:11.470907 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:11.501646 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:11.503145 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:11.538167 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:11.598093 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:11.647076 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:11.669924 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:11.675002 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:11.685072 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:11.687290 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:11.688807 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:11.710461 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:11.752838 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:11.825045 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:11.842367 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:11.886518 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:11.895820 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:11.905927 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:11.923941 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:11.931854 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:11.935073 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:11.948387 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:11.975435 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:11.977828 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:12.031518 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:12.037168 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:12.045523 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:12.046648 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:12.058283 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:12.064561 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:12.100350 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:12.163916 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:12.167821 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:12.170663 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:12.195351 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:12.217887 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:12.295012 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:12.329772 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:12.339756 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:12.363417 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:12.363762 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:12.377547 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:12.383562 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:12.393388 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:12.535969 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:12.537081 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:12.545331 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:12.550343 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:12.558040 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:12.558331 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:12.565297 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:12.577522 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:12.590130 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:12.621680 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:17:12.692562 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:12.729740 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:12.730469 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:12.735599 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:12.744686 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:12.744704 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:12.749725 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:12.757665 16637 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:12.757838 16637 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:12.760748 16637 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:12.762347 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:17:12.762631 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:17:12.764361 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:12.764578 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:12.764675 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:17:12.766659 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:12.766851 16637 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:12.800755 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:12.804237 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:17:12.964071 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:12.967101 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:12.967567 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:12.976320 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:12.993050 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:13.106396 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:13.114341 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:13.143834 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:13.164592 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:13.175446 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:13.176399 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:13.176947 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:13.179360 17149 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:13.179499 17149 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:13.180114 17149 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:13.182886 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:13.183466 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:13.183557 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:13.183945 17149 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:13.209393 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:13.222856 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:13.237885 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:13.279384 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:13.344974 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:13.362905 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:13.371026 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:13.400735 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:13.401464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:13.424398 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:13.583658 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:13.590922 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:13.599851 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:13.612015 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:13.621202 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:13.621860 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:13.639215 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:13.649435 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:13.662189 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:13.663971 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:13.705497 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:13.714794 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:13.722327 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:13.727609 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 14:17:13.794198 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:13.815289 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:13.822407 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:13.831496 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:13.843437 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:13.863621 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:13.874032 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:13.877198 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:13.877962 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:13.878250 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:13.964145 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:14.037952 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:14.112726 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:14.119841 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:14.131609 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:14.171772 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:14.199404 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:17:14.236588 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240503 14:17:14.239835 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:14.243801 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:14.251856 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:14.260360 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:14.276921 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:14.282902 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:14.293637 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:14.300472 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:14.327842 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:14.381232 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:14.403581 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:14.430449 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:14.439961 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:14.459757 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:14.465473 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240503 14:17:14.486197 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:14.494307 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:14.511337 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:14.511925 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:14.535360 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:14.539965 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:14.553383 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:14.593473 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:14.594197 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:14.600081 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:14.605785 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:14.615100 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:14.666805 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:14.684084 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:14.703213 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:14.715489 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:14.715901 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:14.718456 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:14.720517 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:14.785516 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:14.876631 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:14.899313 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:14.906666 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:14.917551 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:14.924528 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:14.924813 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:14.936318 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:14.938989 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:14.966701 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:17:14.995416 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:17:14.996271 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:15.016886 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:15.034056 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:15.095528 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:15.123517 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:15.126101 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:15.138351 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:15.144291 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:15.161135 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:15.198545 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:15.203508 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:15.267439 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:15.276185 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:15.307246 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:15.347775 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:15.351337 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:17:15.364661 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:15.381510 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:15.385679 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:15.517400 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:15.567804 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:15.570343 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:15.584210 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:15.591434 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:15.603802 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:15.617038 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:15.697484 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:15.743404 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:15.757287 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:15.773547 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:15.783475 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:15.791375 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:15.801857 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:15.804972 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:15.807324 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:15.828454 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:15.861361 17782 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:15.861567 17782 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:15.862306 17782 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:17:15.865339 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:15.865658 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:15.865764 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:15.867378 17782 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:15.869287 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:16.055740 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:16.065354 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:16.099493 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:16.142704 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:16.192526 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:16.211501 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:16.215495 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:16.230376 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:16.231494 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:16.253118 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:16.253706 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:16.258656 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:16.283198 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:16.326156 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:16.346940 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:16.370918 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:16.371691 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:16.402917 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:16.419623 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:16.433002 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:16.439404 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:16.445295 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:16.458731 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:16.576807 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:16.582221 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:16.616041 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:16.623714 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:16.629325 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:16.641189 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:16.653486 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:16.654906 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:16.655305 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:16.685173 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:16.689800 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:16.697309 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:16.709347 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:16.811873 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:16.853719 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:16.854446 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:16.867419 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:16.870018 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:16.870980 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:16.885138 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:16.910853 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:16.915017 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:16.945789 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:17.011265 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:17.014336 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:17.067790 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:17.075685 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:17.087102 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:17.099731 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:17.099762 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:17.101866 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:17.107929 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:17.119227 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:17.153784 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:17.163703 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:17.169533 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:17.173133 17834 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:17.173283 17834 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:17.173854 17834 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:17.176429 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:17.180123 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:17.180339 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:17.180454 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:17.180680 17834 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:17.185850 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:17.241539 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:17.279542 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:17.281977 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:17.301961 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:17.302829 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:17.303354 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:17.340860 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:17.391921 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:17.410207 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:17.413194 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:17.531379 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:17.542690 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:17.549624 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:17:17.560470 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:17.572911 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:17.576059 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:17.576709 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:17.659305 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:17.689975 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:17.717515 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:17.741577 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:17.745050 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:17.758298 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:17.759379 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:17.771447 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:17.777513 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:17.780403 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:17.808817 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:17.812451 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:17:17.856720 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:17.973063 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:18.094436 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:18.107236 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:18.139464 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:18.158130 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:18.162196 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:18.165037 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:18.165819 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:18.174069 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:18.179421 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:18.314790 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:18.345410 16637 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:18.345594 16637 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:18.346289 16637 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:18.347450 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:17:18.347735 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:17:18.351240 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:18.351497 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:18.351584 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:18.352010 16637 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:17:18.376976 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:18.390841 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:18.393250 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:18.402761 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:18.403573 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:18.404209 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:18.422080 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:18.504710 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:18.541080 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:18.601824 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:18.613286 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:18.614032 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:18.637751 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:18.643462 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:18.644219 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:18.690465 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:18.701072 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:18.713228 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:18.725720 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:18.738824 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:17:18.832741 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:18.837586 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:18.842875 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:18.855664 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:18.869663 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:18.871922 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:18.885129 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:18.889699 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:18.924497 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:18.933547 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.030596 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:19.047117 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:19.066675 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:19.070881 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:19.084331 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:19.100986 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:19.102968 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.121831 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.231076 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:19.233567 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:19.245626 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:17:19.246556 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:19.267374 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:19.277806 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:19.301002 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:19.312427 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:19.328495 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:19.331202 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240503 14:17:19.333034 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:19.350430 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.365489 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.388114 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.409061 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.421314 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:19.456794 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.502388 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:19.503005 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:19.507089 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.513407 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:17:19.514407 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:17:19.515877 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:19.530511 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:19.541908 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:19.559084 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:19.559587 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:19.569979 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.588773 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:19.631498 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.634207 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.680224 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:19.743664 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:19.749372 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:19.754577 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:19.755000 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:19.760167 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:19.768357 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:19.771960 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:19.773008 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:19.813393 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:19.844913 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:19.883076 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:19.982335 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:20.053208 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:20.110729 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:20.144882 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:20.154632 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 14:17:20.159312 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:20.167829 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:20.199446 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:20.199501 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:20.257526 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:20.272300 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:20.276316 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:17:20.297713 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:20.342164 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:20.352531 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:20.363919 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:20.385847 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:17:20.387938 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:20.398437 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:20.401346 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:20.406678 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:20.407100 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:20.415427 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:20.541114 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:20.585697 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:20.602358 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:20.626281 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:20.626494 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:20.628155 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:20.649775 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:20.695447 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:20.788844 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:20.836396 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:20.837188 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:20.851485 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:20.853323 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:20.875384 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:20.884260 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:20.889292 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:20.926290 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:20.961519 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:17:21.043916 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:21.048168 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:21.054529 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:21.067502 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:21.089480 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:21.112969 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:21.171829 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:21.239638 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:21.259967 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:21.264822 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:21.265379 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:21.268899 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:21.275733 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:21.280424 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:21.281337 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:21.288851 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:21.316444 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:21.325155 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:21.342314 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:21.372169 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:21.434653 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:21.498530 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:21.500115 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:21.521407 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:21.536222 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:21.549763 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:21.562363 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:21.591559 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:21.659548 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:21.671985 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:21.681107 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:21.725558 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:21.731950 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:21.734833 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:21.751282 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:21.756114 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:21.761638 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:21.783968 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:21.799881 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:21.827081 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:21.835055 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:21.878685 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:21.945634 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:21.949651 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:21.972685 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:21.980971 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:22.050334 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:22.068871 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:22.072539 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:22.072680 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:22.101899 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:22.111207 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:22.132678 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:22.142575 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:22.144804 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:22.166040 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:22.170282 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:22.178040 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:22.186290 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:17:22.223346 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:22.246412 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:22.272338 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:22.292447 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:22.304801 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:22.354554 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:22.359370 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:22.362268 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:22.395259 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:22.396855 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:22.420049 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:22.515762 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:22.574010 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:22.612383 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:22.614887 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:22.615013 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:22.615108 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:22.637686 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:22.662063 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:22.716145 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:22.803550 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:22.810169 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:22.815260 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:22.822476 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:22.822829 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:22.837930 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:22.848511 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:22.868063 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:23.011145 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:23.029564 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:23.030588 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:17:23.037539 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:23.046228 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:23.050846 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:23.055680 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:23.059378 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:23.074658 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:23.074895 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:23.106566 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:23.118427 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:23.124604 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:23.237795 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:23.251063 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:23.253299 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:23.253343 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:23.267120 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:23.430718 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:23.440652 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:23.448338 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:23.452383 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:23.459821 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:23.461570 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:23.481976 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:23.488147 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:23.501773 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:23.571065 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:23.622823 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:23.643783 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:23.665177 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:23.675740 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:23.686328 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:23.692684 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:23.699786 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:23.703459 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:23.737218 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:23.742302 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:23.770035 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:23.876327 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:17:23.932374 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:23.975071 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:23.984148 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:24.007530 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:24.074303 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:24.094249 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:24.106674 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:24.106752 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:24.110474 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:24.120994 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:24.137753 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:24.144447 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:24.173063 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:24.222473 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:24.234122 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:24.236827 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:24.261678 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:24.289134 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:24.308346 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:24.349539 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:24.353449 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:17:24.354503 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:24.356923 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:24.372936 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:17:24.373677 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:24.397017 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:24.433418 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:24.439380 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:17:24.440367 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:24.455505 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:24.509424 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:24.520059 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 14:17:24.541491 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:24.570533 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:24.587201 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:24.592550 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:24.597369 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:24.598268 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:24.600904 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:24.616761 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:24.714141 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:24.716236 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:24.730249 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:24.765766 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:24.794201 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:24.804577 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:24.805295 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:24.814239 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:24.817762 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:24.818010 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:24.832867 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:24.836354 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:24.935940 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:24.938794 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:24.995824 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:25.006692 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:25.010893 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:25.029963 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:25.032300 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:25.034438 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:25.047399 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:25.184907 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:17:25.186601 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:25.207280 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:25.207506 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:25.218747 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:25.224648 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:25.230015 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:25.234999 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:25.244342 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:25.259430 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:25.314234 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:25.409516 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:25.415721 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:25.427054 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:25.427994 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:25.433585 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:25.436197 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:25.481099 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:25.567929 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:25.583699 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:25.601663 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:25.622270 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:25.637846 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:25.639817 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:25.656319 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:25.656749 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:25.663033 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:17:25.718478 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:25.819020 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:25.864195 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:25.898548 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:25.918848 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:26.020370 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:26.024626 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:26.033113 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:26.048419 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:26.058177 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:26.063629 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:26.077298 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:26.077513 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:26.101696 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:26.123616 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:26.205332 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:26.217583 19471 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:26.217761 19471 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:26.218690 19471 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:26.219125 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:17:26.219370 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:17:26.220304 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:26.220791 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:26.220927 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:26.221196 19471 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:26.263926 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:26.265645 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:26.284085 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:26.286482 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:26.287003 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:26.307619 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:26.310742 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:26.325127 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:26.457826 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:26.473701 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:26.492995 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:26.497515 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:26.510951 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:26.539113 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:26.540582 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:26.544499 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:26.557395 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:26.642436 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:26.659631 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:26.665894 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:26.673696 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:26.697171 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:26.718880 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:26.720137 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:26.721616 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:26.727998 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:26.740657 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:26.748203 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:26.761700 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:26.811414 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:26.821373 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:26.824846 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:26.880228 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:26.895658 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:26.904116 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:26.905475 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:26.906842 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:26.908207 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:26.927698 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:26.927994 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:17:26.928421 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:26.935451 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:26.938751 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:26.968727 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:26.991436 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:27.030786 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:27.068465 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:27.099838 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:27.101223 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:27.109143 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:27.109534 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:27.115464 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:27.165319 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:27.223496 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:27.255165 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:27.276777 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:27.291926 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:27.292804 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:27.310156 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:27.327345 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:27.328500 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:27.380065 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:27.441622 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:27.444850 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:27.469026 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:27.527424 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:27.534055 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:27.550079 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:27.566982 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:27.570209 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:27.575435 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:27.593613 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:27.607666 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:27.670028 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:27.727324 20038 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:27.727505 20038 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:27.728063 20038 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:27.729312 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:27.729466 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:27.729575 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:27.729775 20038 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:27.819583 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:27.867283 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:27.944474 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:27.953773 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:27.959450 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:27.962890 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:27.967734 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:27.968147 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:27.984982 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:28.015064 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:28.084723 18612 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:28.084905 18612 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:28.085866 18612 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:28.086100 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:17:28.086354 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:17:28.087216 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:28.087656 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:28.087790 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:28.088033 18612 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:17:28.133060 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:28.160635 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:28.161841 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:28.165620 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:28.177564 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:28.185954 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:28.186393 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:28.187862 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:28.197021 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:17:28.227710 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:28.369462 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:28.384066 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:28.400897 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:28.407124 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:28.414899 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:28.416927 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:28.422559 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:28.436228 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:28.449064 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:28.465796 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:28.521737 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:28.613335 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:28.619817 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:28.620851 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:28.638345 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:28.652848 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:28.656934 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:28.686146 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:28.697793 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:28.711431 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:28.760136 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:28.804631 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:28.825695 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:28.826476 2149 maintenance_manager.cc:419] P 13adeda3db494b478178ddd5ee8fdb98: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033382
I20240503 14:17:28.844386 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:28.848959 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:28.857043 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:28.883248 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:28.886626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:28.902336 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:28.906108 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:17:28.938282 2148 maintenance_manager.cc:643] P 13adeda3db494b478178ddd5ee8fdb98: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.111s user 0.041s sys 0.010s Metrics: {"bytes_written":20935,"cfile_init":1,"dirs.queue_time_us":7951,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":38985,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2782,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":43,"thread_start_us":14685,"threads_started":2,"wal-append.queue_time_us":4689}
I20240503 14:17:28.940748 2149 maintenance_manager.cc:419] P 13adeda3db494b478178ddd5ee8fdb98: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7339 bytes on disk
I20240503 14:17:28.942126 2148 maintenance_manager.cc:643] P 13adeda3db494b478178ddd5ee8fdb98: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 14:17:28.946362 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:28.961154 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:28.985577 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:29.058944 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:29.073189 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:29.077896 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:29.095539 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:29.099432 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:29.134176 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.153834 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.168792 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:29.173887 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:29.210644 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:29.297032 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:29.332873 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:29.352525 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:29.355748 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:29.356043 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:29.361147 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:29.363150 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.372889 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:17:29.374303 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:29.377399 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:29.381388 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.385035 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:29.385864 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.385985 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:29.393596 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:29.443298 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:17:29.476514 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.519976 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.526793 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:17:29.574237 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:29.581176 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:29.623246 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:29.631896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:29.637408 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:29.651818 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:29.684242 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:29.716068 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:29.754470 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:29.758096 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:29.774561 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:29.793756 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:29.834740 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:29.906541 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:17:29.984611 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:30.022941 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:30.071331 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:30.078363 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:30.089668 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:30.090143 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:30.091517 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:30.099736 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:30.116117 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:30.215232 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:30.228751 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:30.262635 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:30.280401 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:30.281569 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:30.296778 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:30.297847 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:30.298239 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:30.304522 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:30.312335 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:17:30.429221 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:30.471405 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:30.513944 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:30.527349 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:30.538034 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:30.553859 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:30.554004 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:30.570004 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:30.578883 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:30.682374 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:30.684060 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:30.704937 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:30.735409 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:30.735699 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:30.742192 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:30.759589 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:30.764782 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:30.813326 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:30.833451 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:30.858093 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:30.864712 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:17:30.945160 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:30.949568 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:30.951390 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:30.959393 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:30.964185 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:30.975116 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:31.013146 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:31.015318 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:31.097829 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:31.127810 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:31.156635 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:31.161206 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:31.180250 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:31.188767 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:31.190847 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:31.192728 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:31.193661 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:31.217835 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:31.326872 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:31.340557 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:31.356801 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:31.385605 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:31.398955 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:31.399195 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:31.400233 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:31.405133 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:31.430006 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:31.486691 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:31.505088 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.505249 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.517941 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:31.579522 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:17:31.607838 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:31.622273 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:31.622598 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:31.626101 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:31.644415 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:31.720721 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:31.746794 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:31.747452 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:31.758661 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.766306 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:31.791613 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:31.800791 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:31.840260 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:31.842675 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.862485 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.868999 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.905113 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:31.945071 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:32.035048 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:32.047363 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:32.054687 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:32.058722 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:32.085994 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:32.128177 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:32.129472 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:32.205680 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:32.229166 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:32.264796 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:32.271915 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:32.276041 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:32.276913 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:32.291080 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:32.291115 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:32.291553 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:32.298116 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:32.310127 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:32.340059 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:32.386401 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:32.504812 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:32.517479 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:32.521787 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:32.535279 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:32.546667 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:32.565189 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:32.619881 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:32.667464 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:32.669112 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:32.683926 20625 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:32.684135 20625 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:32.684873 20625 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:32.686201 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:32.686445 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:32.686538 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:32.686761 20625 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:32.689453 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:32.702713 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:32.745157 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:32.762177 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:32.772367 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:32.776151 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:32.790365 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:32.796856 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:32.808843 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:32.939059 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:32.956413 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:32.972913 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:32.984773 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:32.988931 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:32.996326 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:33.013984 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:33.072239 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:33.166775 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:33.178653 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:33.180150 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:33.182929 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:33.188997 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:33.190510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:33.207783 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:33.210299 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:33.218658 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:33.232514 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:33.293256 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:33.302735 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:33.393913 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:33.436120 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:33.439266 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:33.443917 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:33.446349 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:33.464569 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:33.466295 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:33.502568 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:17:33.517669 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:33.591884 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:33.618162 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:33.622229 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:33.628047 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:33.634407 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:33.642943 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:33.645406 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:33.646320 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:33.681207 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:33.686064 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:33.749352 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:33.790160 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:33.838976 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:33.916126 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:33.986889 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:34.012284 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:34.027451 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:34.052713 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:34.059314 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:17:34.067515 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:34.087137 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:34.099974 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:34.102283 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:34.105099 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:34.138927 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:34.140488 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:34.152456 20339 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:34.152664 20339 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:34.153411 20339 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:17:34.156893 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:34.157289 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:34.157531 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:34.157711 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:34.157816 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
W20240503 14:17:34.157938 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:34.158072 20339 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:34.174417 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:34.191159 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:34.195472 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:34.218499 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:34.255477 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:34.288030 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:34.302918 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:34.315093 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:34.317785 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:34.355319 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:34.356349 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:34.360570 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:34.430547 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:34.460568 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:17:34.505677 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:34.509521 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240503 14:17:34.539202 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 14:17:34.562000 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:17:34.564699 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:34.566334 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:34.573982 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:34.575261 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:34.576365 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:34.619570 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:34.680070 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:34.682195 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:34.702369 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:34.726892 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:34.732614 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:34.735352 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:34.757172 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:34.781433 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:34.788892 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:34.803966 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:34.814493 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:34.814843 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:34.905012 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:34.924780 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:35.002305 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:35.008363 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:35.013598 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:35.024684 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:35.033658 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:35.034276 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:35.051818 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:35.058239 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:35.090554 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:35.119845 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:35.150890 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:35.203051 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:35.204458 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:35.207655 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:35.214130 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:35.228868 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:35.265460 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:35.318707 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:35.431936 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:35.432222 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:35.452384 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:35.458091 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:35.462447 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:35.479673 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:35.515292 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 14:17:35.515954 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:35.516939 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:35.541214 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:35.581619 21254 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:35.581800 21254 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:35.582513 21254 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:35.582849 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:17:35.583086 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:17:35.584748 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:35.590991 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:35.591120 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:35.591403 21254 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:35.617442 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:35.633082 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:35.675678 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:35.677758 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:35.697535 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:35.703413 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:35.732203 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:35.732363 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:35.791831 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:35.828297 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:35.842597 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:17:35.886224 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:35.932463 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:36.030056 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.065088 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.066457 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:36.108079 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:36.110848 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:36.117839 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:36.118687 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:36.119434 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:36.139248 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:36.237485 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:36.304673 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:36.305506 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:36.308859 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:36.343293 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:36.343565 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:36.348688 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:36.365594 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:36.382958 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:36.400489 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:36.414292 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.437075 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.472015 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:36.473316 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.485195 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:36.507095 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:36.578038 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:36.586437 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:36.595850 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:36.608536 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:36.621344 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:36.621781 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:36.638654 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:36.670125 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:36.711987 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:36.737638 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:36.752940 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:36.791440 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:36.807641 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:36.809449 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:36.816937 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.827749 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:36.833046 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:36.836009 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:36.851694 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.871671 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:36.883361 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:36.903627 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:36.909675 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:36.988270 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:37.031272 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:37.031881 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:37.037071 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:17:37.048822 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(fe0eebf47c494dad8edf06197cdf15bc): perf score=0.033360
I20240503 14:17:37.051410 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:37.054199 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(fe0eebf47c494dad8edf06197cdf15bc): perf score=0.033370
I20240503 14:17:37.057266 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:37.079114 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:37.081527 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:37.102145 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(dd294a04366145378e3f5e3d306152e3): perf score=0.033382
W20240503 14:17:37.181813 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:37.231606 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:37.235639 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:37.254626 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:37.266827 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(dd294a04366145378e3f5e3d306152e3) complete. Timing: real 0.164s user 0.068s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":28756,"dirs.queue_time_us":9489,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1202,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":49785,"threads_started":3,"wal-append.queue_time_us":9361}
I20240503 14:17:37.268260 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.213s user 0.057s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":346,"dirs.queue_time_us":5722,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":4268,"drs_written":1,"lbm_read_time_us":146,"lbm_reads_lt_1ms":4,"lbm_write_time_us":841,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":22278,"threads_started":3,"wal-append.queue_time_us":6924}
I20240503 14:17:37.270742 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc): 4838 bytes on disk
I20240503 14:17:37.271569 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 14:17:37.272756 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(6c94404d8d4f45d59958ae6feb36fc24): perf score=0.033426
I20240503 14:17:37.274405 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(dd294a04366145378e3f5e3d306152e3): 4923 bytes on disk
I20240503 14:17:37.275233 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(dd294a04366145378e3f5e3d306152e3) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 14:17:37.280289 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(6c94404d8d4f45d59958ae6feb36fc24): perf score=0.033427
I20240503 14:17:37.281420 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.230s user 0.084s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":47457,"dirs.queue_time_us":6780,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":970,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3974,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65948,"threads_started":3,"wal-append.queue_time_us":10432}
W20240503 14:17:37.310680 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:37.310935 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc): 4838 bytes on disk
I20240503 14:17:37.313423 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 14:17:37.386186 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(6c94404d8d4f45d59958ae6feb36fc24) complete. Timing: real 0.113s user 0.044s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":974,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1608,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10668,"threads_started":1,"wal-append.queue_time_us":10556}
I20240503 14:17:37.388056 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(6c94404d8d4f45d59958ae6feb36fc24): 4583 bytes on disk
W20240503 14:17:37.389070 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:37.395701 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(6c94404d8d4f45d59958ae6feb36fc24) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 14:17:37.409400 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:37.440377 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(6c94404d8d4f45d59958ae6feb36fc24) complete. Timing: real 0.160s user 0.061s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1520,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":834,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3556,"threads_started":1,"wal-append.queue_time_us":3602}
I20240503 14:17:37.441960 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(6c94404d8d4f45d59958ae6feb36fc24): 4583 bytes on disk
I20240503 14:17:37.443899 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(6c94404d8d4f45d59958ae6feb36fc24) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240503 14:17:37.460587 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:37.505702 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:37.529299 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:37.548317 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:37.556941 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:37.619953 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:37.634944 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:37.641557 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:37.698993 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:37.718642 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:37.724570 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:37.752138 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:37.803458 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:37.820621 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:37.885838 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:37.905247 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:38.028798 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:38.062322 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:38.102571 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:38.136405 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:38.138216 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:38.198169 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(eb1d30b76c4d43d0a54a49892d09a6f8): perf score=0.033344
I20240503 14:17:38.211293 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:38.224324 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:38.236593 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:38.250013 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:38.260577 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:38.295711 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:38.313457 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:38.318112 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(9974d80e8d31417788e10140de97e722): perf score=0.033378
W20240503 14:17:38.325332 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:17:38.350289 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:38.368541 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:38.375701 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(944d2a983d6247498b02a6e7b7daf746): perf score=0.033395
I20240503 14:17:38.459384 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(eb1d30b76c4d43d0a54a49892d09a6f8) complete. Timing: real 0.261s user 0.083s sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":175575,"compiler_manager_pool.run_cpu_time_us":23,"compiler_manager_pool.run_wall_time_us":24,"dirs.queue_time_us":32598,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":17929,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":48774,"threads_started":2,"wal-append.queue_time_us":16433}
I20240503 14:17:38.461409 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(eb1d30b76c4d43d0a54a49892d09a6f8): 4923 bytes on disk
I20240503 14:17:38.462005 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(eb1d30b76c4d43d0a54a49892d09a6f8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 14:17:38.478745 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(9974d80e8d31417788e10140de97e722) complete. Timing: real 0.159s user 0.068s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18362,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1484,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4187,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25198,"threads_started":2,"wal-append.queue_time_us":7018}
I20240503 14:17:38.480021 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(9974d80e8d31417788e10140de97e722): 4583 bytes on disk
I20240503 14:17:38.480609 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(9974d80e8d31417788e10140de97e722) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 14:17:38.508320 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(944d2a983d6247498b02a6e7b7daf746) complete. Timing: real 0.132s user 0.078s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":6474,"dirs.queue_time_us":6672,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":11484,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":19394,"threads_started":3,"wal-append.queue_time_us":2996}
I20240503 14:17:38.510358 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(944d2a983d6247498b02a6e7b7daf746): 4838 bytes on disk
I20240503 14:17:38.511232 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(944d2a983d6247498b02a6e7b7daf746) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 14:17:38.589080 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:38.594138 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:38.648595 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:38.651414 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:38.669891 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:38.678891 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:38.681273 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:38.692874 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:38.707434 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:38.709935 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:38.717805 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:38.929523 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:38.930128 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:38.942896 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:38.946153 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:38.978019 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:38.993306 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:39.003868 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.006467 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.039988 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:39.069623 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:39.142315 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:39.147112 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:39.153981 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:39.172127 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:39.174535 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:39.175740 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:17:39.179136 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:39.218425 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:39.230409 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.259820 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.295388 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:39.361431 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:39.367978 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:39.369410 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:39.385833 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:39.401201 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:39.402617 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:39.405054 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(8bf4675882e64f35bc3dc9ee06ae366d): perf score=0.033346
I20240503 14:17:39.415282 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:39.423710 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.425032 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:39.427024 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:39.458101 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:39.467955 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(1d584b7026b64ba1b720ed21d7019740): perf score=0.033369
W20240503 14:17:39.480788 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:17:39.481384 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:39.488827 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:39.493340 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(8bf4675882e64f35bc3dc9ee06ae366d) complete. Timing: real 0.088s user 0.043s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":335,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":16536,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":773,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7488,"threads_started":2,"wal-append.queue_time_us":7137}
I20240503 14:17:39.494082 21493 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:39.494238 21493 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:39.494807 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(8bf4675882e64f35bc3dc9ee06ae366d): 4923 bytes on disk
I20240503 14:17:39.494850 21493 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:39.496747 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:39.503248 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(8bf4675882e64f35bc3dc9ee06ae366d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 14:17:39.505107 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(75bafc7675ca45639b696db6e4b7ce36): perf score=0.033372
W20240503 14:17:39.517449 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:39.517596 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:39.517810 21493 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:39.520541 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:39.523466 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:17:39.550758 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(1d584b7026b64ba1b720ed21d7019740) complete. Timing: real 0.082s user 0.035s sys 0.028s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":25320,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1623,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":32098,"threads_started":2,"wal-append.queue_time_us":6723}
I20240503 14:17:39.552505 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(1d584b7026b64ba1b720ed21d7019740): 4583 bytes on disk
I20240503 14:17:39.552690 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(75bafc7675ca45639b696db6e4b7ce36) complete. Timing: real 0.047s user 0.034s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1613,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":264,"threads_started":1,"wal-append.queue_time_us":353}
I20240503 14:17:39.553140 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(1d584b7026b64ba1b720ed21d7019740) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 14:17:39.554193 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(75bafc7675ca45639b696db6e4b7ce36): 4838 bytes on disk
I20240503 14:17:39.554708 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(75bafc7675ca45639b696db6e4b7ce36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 14:17:39.573985 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.623014 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:39.659962 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:17:39.661043 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:39.665426 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:39.675453 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:39.686455 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:39.687892 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:39.694959 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:39.699759 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:39.792958 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:39.839558 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:39.879997 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:39.881793 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:39.901803 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:39.904213 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:39.922694 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:39.942548 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:39.972900 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:39.999352 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:40.030759 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:17:40.057572 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:40.062696 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:40.085912 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:40.109151 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:40.111784 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:40.121582 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:40.133529 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:40.135830 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:40.273712 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:40.274091 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:40.319164 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:40.336388 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:40.368479 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:40.369383 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:40.394613 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:40.415931 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:40.445819 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:40.488003 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(61b63f87fb384bb2a48044bcad87c282): perf score=0.033344
W20240503 14:17:40.520427 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 14:17:40.521138 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(470b425f5cef494ab3334fe2b85e1c4e): perf score=0.033361
W20240503 14:17:40.522568 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:40.545374 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:40.559114 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(fda4460865854505b5220d7ec3b34c57): perf score=0.033364
I20240503 14:17:40.560961 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(61b63f87fb384bb2a48044bcad87c282): perf score=0.033370
W20240503 14:17:40.567130 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:40.586458 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:40.589035 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:40.594403 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(61b63f87fb384bb2a48044bcad87c282) complete. Timing: real 0.106s user 0.049s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11238,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":2475,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":15101,"threads_started":2,"wal-append.queue_time_us":3785}
I20240503 14:17:40.595722 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282): 4837 bytes on disk
I20240503 14:17:40.596361 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 14:17:40.596405 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(470b425f5cef494ab3334fe2b85e1c4e) complete. Timing: real 0.075s user 0.049s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":12235,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1630,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":882,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":20449,"threads_started":2,"wal-append.queue_time_us":8207}
I20240503 14:17:40.597970 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(470b425f5cef494ab3334fe2b85e1c4e): 4923 bytes on disk
I20240503 14:17:40.598606 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(470b425f5cef494ab3334fe2b85e1c4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 14:17:40.615846 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:40.647102 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(61b63f87fb384bb2a48044bcad87c282) complete. Timing: real 0.086s user 0.034s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6311,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":13171,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":973,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":22538,"threads_started":2,"wal-append.queue_time_us":16174}
I20240503 14:17:40.648576 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(470b425f5cef494ab3334fe2b85e1c4e): perf score=0.033388
I20240503 14:17:40.649890 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:40.663697 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:40.667124 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:40.669826 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:40.705044 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(470b425f5cef494ab3334fe2b85e1c4e) complete. Timing: real 0.056s user 0.039s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1462,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3562,"threads_started":1,"wal-append.queue_time_us":3653}
I20240503 14:17:40.706367 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(470b425f5cef494ab3334fe2b85e1c4e): 4923 bytes on disk
I20240503 14:17:40.706971 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(470b425f5cef494ab3334fe2b85e1c4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 14:17:40.707408 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:40.708036 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282): 4837 bytes on disk
I20240503 14:17:40.708595 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 14:17:40.739171 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:40.762347 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:40.788667 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(fda4460865854505b5220d7ec3b34c57) complete. Timing: real 0.229s user 0.053s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1622,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1081,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26925,"threads_started":2,"wal-append.queue_time_us":25310}
I20240503 14:17:40.789852 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(470b425f5cef494ab3334fe2b85e1c4e): perf score=0.033424
W20240503 14:17:40.792194 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:17:40.830729 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(470b425f5cef494ab3334fe2b85e1c4e) complete. Timing: real 0.037s user 0.022s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1457,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":203,"threads_started":1,"wal-append.queue_time_us":306}
I20240503 14:17:40.832057 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(470b425f5cef494ab3334fe2b85e1c4e): 4923 bytes on disk
I20240503 14:17:40.832633 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(470b425f5cef494ab3334fe2b85e1c4e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 14:17:40.833412 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(fda4460865854505b5220d7ec3b34c57): 4583 bytes on disk
I20240503 14:17:40.833868 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(fda4460865854505b5220d7ec3b34c57) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240503 14:17:40.871630 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:40.951972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:40.968528 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:40.974854 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:40.979970 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:41.005658 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:41.034371 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:41.063886 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:41.075299 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:41.094544 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:41.094920 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:41.105279 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:41.294240 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:41.328267 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:41.388824 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:41.392946 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:41.402319 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:41.404347 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:41.409782 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:41.418632 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:41.434494 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:41.453586 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:41.463313 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:41.484210 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:41.486393 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:41.502897 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:41.513765 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:41.604880 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:41.619267 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:41.619472 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:41.622867 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:41.625200 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:41.631898 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:41.633417 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:41.642184 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:41.660343 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:41.677631 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:41.713686 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(65a196c723884d068b3728e876c87eb2): perf score=0.033357
I20240503 14:17:41.761953 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(65a196c723884d068b3728e876c87eb2) complete. Timing: real 0.048s user 0.043s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1822,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6991,"threads_started":2,"wal-append.queue_time_us":6717}
I20240503 14:17:41.763206 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(65a196c723884d068b3728e876c87eb2): 4921 bytes on disk
I20240503 14:17:41.763744 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(65a196c723884d068b3728e876c87eb2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 14:17:41.764772 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(af99824d1853471b867e4ff1f01e9367): perf score=0.033365
W20240503 14:17:41.777937 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:41.796557 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(af99824d1853471b867e4ff1f01e9367) complete. Timing: real 0.032s user 0.026s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1362,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":521,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":404,"threads_started":1,"wal-append.queue_time_us":502}
I20240503 14:17:41.797744 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(af99824d1853471b867e4ff1f01e9367): 4583 bytes on disk
I20240503 14:17:41.799301 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(af99824d1853471b867e4ff1f01e9367) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:17:41.839525 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(65a196c723884d068b3728e876c87eb2): perf score=0.033390
I20240503 14:17:41.851791 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(f139d77b0f0c47809e6408b2771adb73): perf score=0.033385
I20240503 14:17:41.857172 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(af99824d1853471b867e4ff1f01e9367): perf score=0.033383
W20240503 14:17:41.889885 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:41.894834 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:41.895902 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:41.905023 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:41.916734 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:41.927868 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:41.932548 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:41.949280 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:41.962843 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(f139d77b0f0c47809e6408b2771adb73) complete. Timing: real 0.111s user 0.047s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5344,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":9839,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5106,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":42214,"threads_started":2,"wal-append.queue_time_us":36873}
I20240503 14:17:41.964176 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(f139d77b0f0c47809e6408b2771adb73): 4838 bytes on disk
I20240503 14:17:41.965070 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(f139d77b0f0c47809e6408b2771adb73) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240503 14:17:41.968032 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:41.972620 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:41.988263 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(af99824d1853471b867e4ff1f01e9367) complete. Timing: real 0.131s user 0.038s sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6549,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1854,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":872,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31976,"threads_started":2,"wal-append.queue_time_us":24384}
I20240503 14:17:41.989846 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(af99824d1853471b867e4ff1f01e9367): 4583 bytes on disk
I20240503 14:17:41.990568 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(af99824d1853471b867e4ff1f01e9367) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 14:17:41.991501 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(65a196c723884d068b3728e876c87eb2) complete. Timing: real 0.152s user 0.043s sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":29887,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1891,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":998,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":32339,"threads_started":2,"wal-append.queue_time_us":2301}
I20240503 14:17:41.994573 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(f139d77b0f0c47809e6408b2771adb73): perf score=0.033429
W20240503 14:17:42.018860 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:42.020588 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:42.031647 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:42.035712 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:42.039620 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:42.073058 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(f139d77b0f0c47809e6408b2771adb73) complete. Timing: real 0.078s user 0.052s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1395,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":205,"threads_started":1,"wal-append.queue_time_us":861}
I20240503 14:17:42.074447 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(65a196c723884d068b3728e876c87eb2): 4921 bytes on disk
I20240503 14:17:42.074951 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(65a196c723884d068b3728e876c87eb2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240503 14:17:42.076022 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(f139d77b0f0c47809e6408b2771adb73): 4838 bytes on disk
I20240503 14:17:42.076468 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(f139d77b0f0c47809e6408b2771adb73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 14:17:42.109304 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:42.109825 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:42.173478 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:17:42.176669 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:42.184612 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:42.193814 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:42.194732 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:42.196496 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:42.215215 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:42.281920 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:42.341267 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:42.388267 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:42.388500 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:42.409126 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:42.413919 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:42.423008 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:42.423377 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:42.445931 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:42.459800 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:42.517292 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:42.538338 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:42.552124 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:42.607511 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:42.624357 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:42.632688 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:42.641561 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:42.648630 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:42.652381 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:42.722014 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(6325a400641b4d25b95592abb4af7d2d): perf score=0.033355
I20240503 14:17:42.745829 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(b359fb8d9afc4c9c856897a722100304): perf score=0.033358
I20240503 14:17:42.775763 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(6325a400641b4d25b95592abb4af7d2d) complete. Timing: real 0.053s user 0.040s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":474,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13218,"threads_started":2,"wal-append.queue_time_us":12919}
I20240503 14:17:42.776825 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(dc88c5dc94844f1c8400c31ad1165a32): perf score=0.033368
I20240503 14:17:42.802657 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(b359fb8d9afc4c9c856897a722100304) complete. Timing: real 0.057s user 0.040s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":349,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1317,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3567,"threads_started":2,"wal-append.queue_time_us":3433}
I20240503 14:17:42.804159 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(b359fb8d9afc4c9c856897a722100304): 4838 bytes on disk
I20240503 14:17:42.804333 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(dc88c5dc94844f1c8400c31ad1165a32): perf score=0.033373
I20240503 14:17:42.804708 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(b359fb8d9afc4c9c856897a722100304) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 14:17:42.828014 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(dc88c5dc94844f1c8400c31ad1165a32) complete. Timing: real 0.051s user 0.027s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1651,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":839,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8534,"threads_started":1,"wal-append.queue_time_us":8638}
I20240503 14:17:42.829689 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(6325a400641b4d25b95592abb4af7d2d): 4923 bytes on disk
I20240503 14:17:42.830332 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(6325a400641b4d25b95592abb4af7d2d): perf score=0.033380
I20240503 14:17:42.830915 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(6325a400641b4d25b95592abb4af7d2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 14:17:42.840430 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(dc88c5dc94844f1c8400c31ad1165a32): 4583 bytes on disk
W20240503 14:17:42.841744 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:42.842481 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(dc88c5dc94844f1c8400c31ad1165a32) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240503 14:17:42.844811 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:42.874234 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:42.884645 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(6325a400641b4d25b95592abb4af7d2d) complete. Timing: real 0.054s user 0.034s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4745,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":5152,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5688,"threads_started":2,"wal-append.queue_time_us":912}
I20240503 14:17:42.885638 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(6325a400641b4d25b95592abb4af7d2d): 4923 bytes on disk
I20240503 14:17:42.885664 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(dc88c5dc94844f1c8400c31ad1165a32) complete. Timing: real 0.081s user 0.042s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3853,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1379,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":789,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":16349,"threads_started":2,"wal-append.queue_time_us":12679}
I20240503 14:17:42.886266 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(6325a400641b4d25b95592abb4af7d2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240503 14:17:42.886896 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(dc88c5dc94844f1c8400c31ad1165a32): 4583 bytes on disk
I20240503 14:17:42.887512 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(dc88c5dc94844f1c8400c31ad1165a32) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240503 14:17:42.936555 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:42.946096 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:42.959746 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:42.964890 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:42.971848 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:42.979432 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:42.980388 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:42.987828 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:43.082459 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:43.134302 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:43.159029 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:43.168622 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:43.174384 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:43.174760 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:43.191080 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:43.199304 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:43.205041 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:43.319836 21955 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:43.319969 21955 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:43.320477 21955 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:43.321547 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:43.321977 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:43.322080 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:43.322274 21955 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:43.358224 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:17:43.384658 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:43.388167 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:43.525530 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:43.526679 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:43.581986 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:43.582839 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:43.599671 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:43.600431 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:43.623196 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:43.625124 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:43.662133 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:43.688711 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:43.756971 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:43.772603 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:43.789774 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:43.803653 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:43.813230 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:43.817023 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:43.819429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:43.833914 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:43.849654 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:43.863991 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:43.874281 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:43.892004 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(abdff0b719a342cfbde746751bd2a89d): perf score=0.033349
I20240503 14:17:43.892074 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(b112f8fb89c7404b87a4c6e3d72fdcfd): perf score=0.033349
W20240503 14:17:43.919581 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:43.950314 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(abdff0b719a342cfbde746751bd2a89d) complete. Timing: real 0.058s user 0.020s sys 0.036s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":12489,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":2575,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17789,"threads_started":2,"wal-append.queue_time_us":5465}
I20240503 14:17:43.951738 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(abdff0b719a342cfbde746751bd2a89d): 4583 bytes on disk
I20240503 14:17:43.952361 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(abdff0b719a342cfbde746751bd2a89d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 14:17:43.963584 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(b112f8fb89c7404b87a4c6e3d72fdcfd) complete. Timing: real 0.071s user 0.057s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":333,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1876,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1107,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":445,"threads_started":2,"wal-append.queue_time_us":294}
I20240503 14:17:43.964692 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(b112f8fb89c7404b87a4c6e3d72fdcfd): 4838 bytes on disk
I20240503 14:17:43.965229 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(b112f8fb89c7404b87a4c6e3d72fdcfd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 14:17:44.004783 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:44.028837 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.041111 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:44.054129 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:44.058557 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:44.060463 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(abdff0b719a342cfbde746751bd2a89d): perf score=0.033396
I20240503 14:17:44.073005 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:44.079309 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:44.087826 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:44.107519 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:44.147637 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(abdff0b719a342cfbde746751bd2a89d) complete. Timing: real 0.086s user 0.041s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":24436,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":7677,"drs_written":1,"lbm_read_time_us":157,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31818,"threads_started":2,"wal-append.queue_time_us":7134}
I20240503 14:17:44.149026 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(78edc267e9f44d82b7dbb663628359fe): perf score=0.033416
W20240503 14:17:44.161141 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.170459 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.177721 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:17:44.178627 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:44.192816 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(78edc267e9f44d82b7dbb663628359fe) complete. Timing: real 0.044s user 0.030s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":337,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":129408,"thread_start_us":1735,"threads_started":1,"wal-append.queue_time_us":1794}
I20240503 14:17:44.194101 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(b112f8fb89c7404b87a4c6e3d72fdcfd): perf score=0.033426
W20240503 14:17:44.195300 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:44.224362 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:44.242750 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(b112f8fb89c7404b87a4c6e3d72fdcfd) complete. Timing: real 0.048s user 0.041s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":110,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":5045,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":240,"threads_started":1,"wal-append.queue_time_us":355}
W20240503 14:17:44.243553 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:44.244634 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(78edc267e9f44d82b7dbb663628359fe): 4923 bytes on disk
I20240503 14:17:44.245278 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(78edc267e9f44d82b7dbb663628359fe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240503 14:17:44.246461 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(b112f8fb89c7404b87a4c6e3d72fdcfd): 4838 bytes on disk
I20240503 14:17:44.247009 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(b112f8fb89c7404b87a4c6e3d72fdcfd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 14:17:44.248082 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(abdff0b719a342cfbde746751bd2a89d): 4583 bytes on disk
I20240503 14:17:44.248802 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(abdff0b719a342cfbde746751bd2a89d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240503 14:17:44.250625 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.263562 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:44.322028 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:44.354677 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:44.369133 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:44.379612 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:44.386490 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:44.396556 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:44.402758 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:44.424665 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.427690 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:44.476644 22405 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:44.476821 22405 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:44.477648 22405 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:44.477977 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:17:44.478184 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:17:44.479102 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:44.479324 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:44.479429 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:44.479648 22405 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:44.507813 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:17:44.550186 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:44.556218 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:44.560518 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240503 14:17:44.568596 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:44.582769 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:44.589249 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:44.590919 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:44.599906 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:44.630949 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:44.639827 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:44.660672 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 14:17:44.666199 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.745939 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:44.757572 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:44.757843 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:44.767261 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:44.811975 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:44.812642 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:44.817525 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:44.820844 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:44.841742 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:44.848106 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:44.907248 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:44.957569 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:17:44.957998 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(ff59573e2c8f472585b96ca0a0fd75b1): perf score=0.033335
W20240503 14:17:44.969034 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:44.970765 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(832f07a6b12740d08fa67d653b3818bf): perf score=0.033338
W20240503 14:17:44.976230 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:45.002875 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(a9a474791f9b4585aa0114162705c96f): perf score=0.033347
I20240503 14:17:45.034487 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(832f07a6b12740d08fa67d653b3818bf) complete. Timing: real 0.064s user 0.044s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6830,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":8594,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":810,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17062,"threads_started":2,"wal-append.queue_time_us":10181}
I20240503 14:17:45.036118 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(832f07a6b12740d08fa67d653b3818bf): 4838 bytes on disk
I20240503 14:17:45.038709 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(832f07a6b12740d08fa67d653b3818bf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 14:17:45.048779 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:45.061734 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:45.082469 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(a9a474791f9b4585aa0114162705c96f) complete. Timing: real 0.079s user 0.035s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":410,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1946,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":808,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10015,"threads_started":2,"wal-append.queue_time_us":9169}
I20240503 14:17:45.083988 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(a9a474791f9b4585aa0114162705c96f): 4923 bytes on disk
I20240503 14:17:45.085157 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(a9a474791f9b4585aa0114162705c96f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 14:17:45.086942 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(ff59573e2c8f472585b96ca0a0fd75b1): perf score=0.033368
I20240503 14:17:45.096036 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(ff59573e2c8f472585b96ca0a0fd75b1) complete. Timing: real 0.138s user 0.056s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3020,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":5823,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1015,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25468,"threads_started":2,"wal-append.queue_time_us":12465}
I20240503 14:17:45.097769 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(ff59573e2c8f472585b96ca0a0fd75b1): 4583 bytes on disk
I20240503 14:17:45.098251 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:45.102523 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(832f07a6b12740d08fa67d653b3818bf): perf score=0.033380
I20240503 14:17:45.121162 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(ff59573e2c8f472585b96ca0a0fd75b1) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1227,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 14:17:45.126127 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(a9a474791f9b4585aa0114162705c96f): perf score=0.033379
I20240503 14:17:45.129963 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:45.131430 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:45.160077 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:45.182662 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(ff59573e2c8f472585b96ca0a0fd75b1) complete. Timing: real 0.095s user 0.034s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":6144,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11588,"threads_started":1,"wal-append.queue_time_us":10524}
I20240503 14:17:45.184268 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(ff59573e2c8f472585b96ca0a0fd75b1): 4583 bytes on disk
I20240503 14:17:45.185859 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(ff59573e2c8f472585b96ca0a0fd75b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 14:17:45.204411 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(a9a474791f9b4585aa0114162705c96f) complete. Timing: real 0.078s user 0.039s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":706,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10672,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":270,"threads_started":1,"wal-append.queue_time_us":371}
I20240503 14:17:45.205986 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(a9a474791f9b4585aa0114162705c96f): 4923 bytes on disk
I20240503 14:17:45.206619 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(a9a474791f9b4585aa0114162705c96f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:17:45.215930 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(832f07a6b12740d08fa67d653b3818bf) complete. Timing: real 0.113s user 0.058s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5425,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1402,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":723,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6772,"threads_started":2,"wal-append.queue_time_us":1233}
I20240503 14:17:45.217176 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(ff59573e2c8f472585b96ca0a0fd75b1): perf score=0.033407
W20240503 14:17:45.259970 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:45.267505 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(ff59573e2c8f472585b96ca0a0fd75b1) complete. Timing: real 0.049s user 0.032s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1337,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":743,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6416,"threads_started":1,"wal-append.queue_time_us":6297}
I20240503 14:17:45.269055 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(832f07a6b12740d08fa67d653b3818bf): 4838 bytes on disk
I20240503 14:17:45.269616 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(832f07a6b12740d08fa67d653b3818bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:17:45.270722 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(ff59573e2c8f472585b96ca0a0fd75b1): 4583 bytes on disk
I20240503 14:17:45.271127 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(ff59573e2c8f472585b96ca0a0fd75b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240503 14:17:45.293678 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:45.381686 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:45.395116 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:45.398165 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:45.407358 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:45.411489 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:45.415517 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:45.423993 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:45.424933 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:45.517789 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:45.544345 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:45.554422 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:17:45.555290 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:45.617987 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:45.650980 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:45.716238 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:45.794651 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:17:45.809405 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:45.826772 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:45.826869 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:45.827459 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:45.829859 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:45.834951 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:45.846314 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:45.982365 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:46.016297 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:46.022172 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:46.028925 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:46.034683 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:46.060690 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:46.073987 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:46.075361 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:46.089401 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:46.161846 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:46.230532 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:46.232229 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:46.233310 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:46.250495 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:46.257535 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:46.258558 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:46.262416 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:46.276191 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:46.286301 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:46.309619 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:46.375149 22633 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:46.375396 22633 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:46.376294 22633 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:46.376725 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:17:46.376950 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:17:46.378230 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:46.378883 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:46.378990 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:46.379276 22633 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:46.401389 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:46.476711 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:46.481529 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:46.482301 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:46.505681 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:46.514485 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:46.515347 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:46.540030 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:46.540728 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:46.578039 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:46.579133 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:46.600675 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:46.613773 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:46.619438 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:46.670680 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:46.676587 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:46.727272 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:46.728149 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:46.750331 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:46.755921 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:46.758769 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:46.765619 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:46.784173 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:46.785107 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:46.836951 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:46.847160 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:46.858716 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:46.915504 22405 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:46.915732 22405 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:46.916788 22405 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:17:46.921325 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:46.921550 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:46.921674 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:46.921943 22405 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:17:46.922881 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:46.928452 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:46.931672 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:46.943609 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:46.958547 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:46.961956 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:46.987754 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:46.992914 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:46.993149 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:47.033596 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(9ae190d0f4d843c990e6fdadcac677ed): perf score=0.033348
I20240503 14:17:47.048930 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(aba6537cdb87400cb8b108b1fbfa6c0f): perf score=0.033346
W20240503 14:17:47.078119 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:47.088450 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(9ae190d0f4d843c990e6fdadcac677ed) complete. Timing: real 0.055s user 0.038s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1617,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":1523,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":799,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5798,"threads_started":2,"wal-append.queue_time_us":4390}
I20240503 14:17:47.093262 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(9ae190d0f4d843c990e6fdadcac677ed): 4583 bytes on disk
I20240503 14:17:47.093952 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(9ae190d0f4d843c990e6fdadcac677ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 14:17:47.094976 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(059a4a35aa94412994bb431342ba0753): perf score=0.033359
W20240503 14:17:47.111436 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:47.123116 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(aba6537cdb87400cb8b108b1fbfa6c0f) complete. Timing: real 0.074s user 0.059s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1369,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":833,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5105,"threads_started":2,"wal-append.queue_time_us":4958}
I20240503 14:17:47.124730 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(aba6537cdb87400cb8b108b1fbfa6c0f): 4921 bytes on disk
I20240503 14:17:47.125347 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(aba6537cdb87400cb8b108b1fbfa6c0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240503 14:17:47.161159 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:47.164194 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(059a4a35aa94412994bb431342ba0753) complete. Timing: real 0.069s user 0.056s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1227,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":250,"threads_started":1,"wal-append.queue_time_us":381}
I20240503 14:17:47.165235 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(059a4a35aa94412994bb431342ba0753): 4836 bytes on disk
I20240503 14:17:47.165843 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(059a4a35aa94412994bb431342ba0753) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 14:17:47.196873 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:47.197348 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(059a4a35aa94412994bb431342ba0753): perf score=0.033396
I20240503 14:17:47.202668 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:47.217173 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(aba6537cdb87400cb8b108b1fbfa6c0f): perf score=0.033399
I20240503 14:17:47.221973 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:47.222240 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:47.222863 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:47.234282 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:47.261106 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:47.261631 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:47.281599 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:47.322855 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:47.333853 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(aba6537cdb87400cb8b108b1fbfa6c0f) complete. Timing: real 0.116s user 0.054s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8434,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":15448,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":996,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9024,"threads_started":2,"wal-append.queue_time_us":565}
I20240503 14:17:47.336230 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(aba6537cdb87400cb8b108b1fbfa6c0f): 4921 bytes on disk
I20240503 14:17:47.336854 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(aba6537cdb87400cb8b108b1fbfa6c0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 14:17:47.339030 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(059a4a35aa94412994bb431342ba0753) complete. Timing: real 0.141s user 0.050s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":756,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":862,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8661,"threads_started":2,"wal-append.queue_time_us":5946}
I20240503 14:17:47.340428 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(059a4a35aa94412994bb431342ba0753): 4836 bytes on disk
I20240503 14:17:47.341038 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(059a4a35aa94412994bb431342ba0753) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 14:17:47.342136 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(9ae190d0f4d843c990e6fdadcac677ed): perf score=0.033425
W20240503 14:17:47.367805 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:47.379793 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(9ae190d0f4d843c990e6fdadcac677ed) complete. Timing: real 0.037s user 0.028s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1447,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1712,"threads_started":1,"wal-append.queue_time_us":1813}
I20240503 14:17:47.381696 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(9ae190d0f4d843c990e6fdadcac677ed): 4583 bytes on disk
I20240503 14:17:47.382241 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(9ae190d0f4d843c990e6fdadcac677ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 14:17:47.385735 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:47.396059 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:47.439798 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:47.496512 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:47.504009 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:47.509821 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:47.516654 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:47.530081 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:47.538405 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:47.696614 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:47.753554 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:17:47.766037 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:47.791023 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:47.814774 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:47.827082 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:47.911777 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:47.912983 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:47.932761 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:47.934351 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:47.943542 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:47.947830 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:47.955317 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:48.014266 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:48.081277 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:48.105077 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:48.118149 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:48.128159 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:48.133354 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:48.137394 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(57c4a133b5c646f78b07f37dc85904f6): perf score=0.033341
I20240503 14:17:48.164147 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:48.170939 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling FlushMRSOp(5f631c7713d24f4caf7550bb05147615): perf score=0.033349
W20240503 14:17:48.204306 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:48.236603 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: FlushMRSOp(5f631c7713d24f4caf7550bb05147615) complete. Timing: real 0.065s user 0.045s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1131,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":2374,"drs_written":1,"lbm_read_time_us":140,"lbm_reads_lt_1ms":4,"lbm_write_time_us":990,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":11383,"threads_started":2,"wal-append.queue_time_us":10462}
I20240503 14:17:48.238050 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(5f631c7713d24f4caf7550bb05147615): 4583 bytes on disk
I20240503 14:17:48.238667 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(5f631c7713d24f4caf7550bb05147615) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:17:48.242136 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(57c4a133b5c646f78b07f37dc85904f6) complete. Timing: real 0.105s user 0.039s sys 0.036s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":23844,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":4235,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":30432,"threads_started":2,"wal-append.queue_time_us":6525}
I20240503 14:17:48.243155 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(57c4a133b5c646f78b07f37dc85904f6): 4838 bytes on disk
I20240503 14:17:48.243773 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(57c4a133b5c646f78b07f37dc85904f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 14:17:48.298398 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:48.311671 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:48.342104 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling FlushMRSOp(57c4a133b5c646f78b07f37dc85904f6): perf score=0.033393
W20240503 14:17:48.345563 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:17:48.381804 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling FlushMRSOp(5f631c7713d24f4caf7550bb05147615): perf score=0.033402
I20240503 14:17:48.394201 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: FlushMRSOp(57c4a133b5c646f78b07f37dc85904f6) complete. Timing: real 0.052s user 0.033s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8944,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":9968,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1000,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9257,"threads_started":2,"wal-append.queue_time_us":305}
I20240503 14:17:48.395810 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(57c4a133b5c646f78b07f37dc85904f6): 4838 bytes on disk
I20240503 14:17:48.396839 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(57c4a133b5c646f78b07f37dc85904f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240503 14:17:48.397257 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:48.399893 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:48.410017 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:48.432852 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:48.445202 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:48.452723 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:48.473446 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:48.483462 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: FlushMRSOp(5f631c7713d24f4caf7550bb05147615) complete. Timing: real 0.101s user 0.079s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":346,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":20885,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1065,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14166,"threads_started":2,"wal-append.queue_time_us":13622}
I20240503 14:17:48.485996 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(5f631c7713d24f4caf7550bb05147615): 4583 bytes on disk
I20240503 14:17:48.489686 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(5f631c7713d24f4caf7550bb05147615) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 14:17:48.490425 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:48.493181 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:48.560946 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:48.565310 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:48.677105 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:48.692287 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:48.701911 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:48.702610 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:48.703332 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:48.723726 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:48.737869 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:48.738454 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:48.738639 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:48.797816 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:48.807078 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:48.845875 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:48.907305 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:48.936698 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:48.938794 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:48.942994 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:48.948288 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:48.962391 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:48.994813 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:48.998051 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:49.001484 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:49.008005 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:49.020839 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:49.033361 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:49.121976 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:49.157867 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:49.158788 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:49.160763 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:49.173463 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:49.173890 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:49.177721 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:49.193234 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:49.242415 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:49.324745 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:49.373497 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:49.380661 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:49.395565 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:49.400260 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:49.406296 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:49.416844 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:49.425446 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:17:49.435214 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:49.457576 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:49.486469 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:49.517835 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:49.528934 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:17:49.535655 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:49.537047 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:49.582047 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:49.588286 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:49.597510 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:49.601130 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:49.607678 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:17:49.608484 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:49.616747 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:49.617007 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:49.619887 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:49.625171 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:49.636610 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:49.676203 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:17:49.780457 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:49.891813 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:17:49.986251 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:49.991406 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:50.003701 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:50.003803 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:50.030417 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:50.038391 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:50.046268 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:50.063081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:50.065968 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:50.077191 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:50.078291 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:50.095047 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:50.115862 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:50.162392 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:50.219460 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:50.227320 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:50.233337 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:50.260146 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:50.266798 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:50.301378 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:50.381793 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:50.449864 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:50.483323 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:50.507730 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:50.518244 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:50.519049 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:50.527664 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:50.527952 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:50.528934 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:50.532250 23160 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:50.532416 23160 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:50.533015 23160 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:50.544430 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:50.546245 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:50.546370 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:50.546820 23160 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:50.548408 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:50.579568 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:17:50.580000 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:50.580478 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:50.712942 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:50.748452 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:50.749024 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:50.750797 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:50.751412 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:50.843271 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:50.910463 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:17:50.924271 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:50.945935 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:50.953490 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:50.955893 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:50.960415 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:50.977298 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:50.990100 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:51.004441 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:51.039429 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:51.043385 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:51.109951 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:51.154924 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:51.161665 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:51.167054 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:51.199111 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:51.200156 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:51.204022 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:51.221685 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:51.233566 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:51.241698 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:51.266577 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:51.285168 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:51.295910 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:51.340116 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:51.389955 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:51.393070 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:51.393894 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:51.400360 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:51.414860 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:51.416100 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:51.438063 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:51.479125 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:51.487421 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:51.495299 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:51.500367 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:51.521658 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:51.578814 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:51.591612 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:51.594563 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:51.596472 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:51.623483 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:51.623873 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:51.642771 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:51.698765 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:51.711846 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:51.799424 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:51.831077 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:51.915853 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:51.932219 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:51.959954 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:51.965849 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:52.001588 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:52.027748 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:52.042897 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:52.047799 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:52.048694 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:52.060245 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:52.089201 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:52.129989 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:52.221725 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:52.226323 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:52.239132 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:17:52.260576 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:52.264427 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:17:52.289559 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:52.309412 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:52.356873 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:17:52.359025 23543 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:52.359206 23543 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:52.359835 23543 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:52.361677 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:52.361805 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:52.361930 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:52.362109 23543 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:52.413902 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:52.429214 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:52.483129 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:52.489506 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:52.490088 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:52.513176 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:52.519919 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:52.530860 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:52.540971 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:52.598520 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:52.663880 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:52.673242 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:52.699867 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:52.709745 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:52.725005 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:52.730587 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:52.732403 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:52.737040 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:52.737779 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:52.754940 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:52.783985 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:52.915251 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:52.931412 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:52.939764 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:52.940438 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:52.941270 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:52.956105 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:52.976567 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:52.984318 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:53.001394 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:53.074838 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:53.121958 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:53.139290 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:17:53.174799 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:53.175339 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:53.176837 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:53.182785 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:53.198953 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:53.241282 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:53.293107 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:17:53.340091 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:53.399699 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:53.408638 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:53.412220 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:53.412659 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:53.431102 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:53.448740 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:17:53.491456 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:17:53.495319 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:17:53.529552 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:53.556609 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:53.572721 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:53.608522 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:53.618132 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:53.625957 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:53.630690 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:53.643146 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:53.714339 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:53.739538 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:17:53.743570 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:53.767467 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:53.796599 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:53.800727 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:17:53.809034 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:53.846611 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:53.862205 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:53.888228 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:53.939168 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:54.026450 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:54.050711 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:54.058990 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:54.062937 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:54.075131 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:54.078125 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:54.091257 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:54.092599 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:54.137614 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.201709 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:54.246418 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:17:54.264905 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:54.271857 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:54.288303 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:54.289302 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:54.297227 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:17:54.316020 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:54.344820 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:54.408380 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.422425 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:54.491297 23623 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:54.491490 23623 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:54.492267 23623 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:17:54.495572 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:54.495857 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:54.495942 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:54.497231 23623 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:54.499644 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.522862 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.526976 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:54.529882 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:54.531848 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:54.547792 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:54.553298 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:54.559937 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:54.560567 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:54.578467 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:54.605401 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 14:17:54.628935 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 14:17:54.646198 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:17:54.663353 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.694908 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:17:54.730715 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:17:54.765182 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:17:54.789812 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:54.796655 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:54.797915 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:54.808368 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:54.808964 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:54.811914 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:54.859496 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:54.884954 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.894491 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:54.914829 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:54.963315 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:17:55.014397 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:55.020874 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:55.040084 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:55.043924 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:55.050374 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:55.065719 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:55.092630 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:55.100481 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:55.177058 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:55.219767 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:55.222301 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:55.230492 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:55.235396 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:55.252010 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:55.257618 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:55.282045 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:55.298169 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:55.314199 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:55.355149 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:55.431506 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:17:55.447252 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:55.451756 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:55.467347 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:55.482180 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:55.490749 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:55.497213 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:55.527164 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:55.536031 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:17:55.547410 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:17:55.685549 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:17:55.686825 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:17:55.706619 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:55.708411 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:55.735397 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:55.736035 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:55.756034 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:55.837460 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:55.844092 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:55.860905 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:55.948846 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:55.989024 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:17:56.020589 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:17:56.030248 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:17:56.048731 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:17:56.122503 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:56.130110 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:56.135309 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:56.141273 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:56.144400 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:17:56.146042 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:56.152961 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:56.153283 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:56.158818 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:56.261411 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:56.307279 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:56.352020 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:56.356809 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:56.367004 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:56.373183 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:56.374377 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:56.400499 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:56.411156 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:56.425803 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:56.452390 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:56.460036 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:56.501274 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:56.509912 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:56.556465 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:56.605971 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:56.607941 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:56.611739 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:56.620147 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:56.639833 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:17:56.665854 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:56.686630 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:56.775313 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:56.778522 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:17:56.813761 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:56.837567 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:56.855544 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:56.857828 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:56.861477 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:56.865388 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:56.894199 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:56.902223 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:56.922408 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:56.925032 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:56.941982 24362 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:56.942183 24362 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:56.943034 24362 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:56.943946 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:17:56.944182 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:17:56.946282 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:56.946990 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:56.947109 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:56.947420 24362 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:56.961596 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:57.030988 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:57.049620 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:57.061717 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:57.062763 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:57.082660 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:57.093241 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:57.124719 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:57.139410 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:57.198340 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:17:57.257252 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:57.267539 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:57.284919 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:17:57.296236 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:57.325527 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:57.325947 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:57.326398 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:57.326484 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:57.372400 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:57.431859 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:57.519085 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:57.540958 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:57.551574 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:57.560578 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:57.563818 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:57.564872 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:57.576028 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:57.598733 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:57.658941 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:17:57.723469 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:17:57.762272 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:57.767267 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:57.775590 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:57.787067 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:57.790807 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:57.802107 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:17:57.803514 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:17:57.805022 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:57.843534 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:57.878134 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:57.882411 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:57.922569 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:57.962857 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:17:58.019519 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:58.042855 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:58.137836 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:17:58.177481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:58.189838 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:58.196399 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:58.197746 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:58.215772 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:58.244146 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:58.330857 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:58.342739 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:17:58.386533 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:58.406997 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:58.409125 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:58.412495 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:58.413952 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:58.432018 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:58.436072 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:58.450978 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:17:58.501791 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:17:58.513085 23543 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:58.513263 23543 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:17:58.513972 23543 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:17:58.515394 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:58.516139 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:58.516389 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:58.516479 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:58.516716 23543 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:58.523401 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:58.529026 24402 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:17:58.529206 24402 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:58.529950 24402 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:17:58.531525 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:58.533996 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:58.534138 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:58.534301 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 1 yes votes; 3 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:17:58.534556 24402 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:58.567588 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:17:58.605348 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:58.609913 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:17:58.615854 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:17:58.644778 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:58.648025 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:17:58.648166 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:58.650874 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:58.654127 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:17:58.666539 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:58.672039 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:58.703687 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:58.716969 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:58.727823 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:58.808678 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:58.833446 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:58.843385 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:58.863628 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:58.867220 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:58.867587 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:58.956045 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:58.999207 24402 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:17:58.999431 24402 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:17:59.000267 24402 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:17:59.000595 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:17:59.000846 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:17:59.003518 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:59.004163 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:59.004325 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:17:59.004673 24402 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:17:59.029735 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:17:59.052896 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:17:59.059440 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:59.067471 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:59.073954 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:59.095384 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:59.098134 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:17:59.144584 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:59.172137 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(eb1d30b76c4d43d0a54a49892d09a6f8): perf score=0.033350
W20240503 14:17:59.184278 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:17:59.195114 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:59.197896 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:59.237890 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:59.259984 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:59.282598 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:17:59.284224 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(eb1d30b76c4d43d0a54a49892d09a6f8) complete. Timing: real 0.111s user 0.094s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":382,"dirs.queue_time_us":2374,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":846,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1347,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2748,"threads_started":3,"wal-append.queue_time_us":319}
I20240503 14:17:59.285718 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(eb1d30b76c4d43d0a54a49892d09a6f8): 4923 bytes on disk
I20240503 14:17:59.286417 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(eb1d30b76c4d43d0a54a49892d09a6f8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 14:17:59.311671 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:17:59.320798 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:59.407124 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:59.411978 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:59.416625 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:59.427629 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:59.468748 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:17:59.486910 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:17:59.506506 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:17:59.523365 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:17:59.576361 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:17:59.638543 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:17:59.640493 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:17:59.672636 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:17:59.681627 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:59.685469 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:17:59.735915 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:59.741374 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:17:59.748122 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:17:59.749512 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 128 similar messages]
I20240503 14:17:59.761950 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:17:59.773823 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:17:59.782595 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:17:59.815658 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:17:59.828529 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:17:59.843909 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:17:59.922973 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:17:59.943152 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:17:59.970996 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:17:59.996961 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:00.001520 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:00.015579 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:00.019147 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:00.022383 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:00.025811 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:00.035394 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:00.040364 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(6c94404d8d4f45d59958ae6feb36fc24): perf score=0.033380
I20240503 14:18:00.111745 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(6c94404d8d4f45d59958ae6feb36fc24) complete. Timing: real 0.071s user 0.043s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":293,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1867,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":622,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":428,"threads_started":2,"wal-append.queue_time_us":358}
I20240503 14:18:00.112955 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(6c94404d8d4f45d59958ae6feb36fc24): 4583 bytes on disk
I20240503 14:18:00.113552 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(6c94404d8d4f45d59958ae6feb36fc24) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240503 14:18:00.161180 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:00.175848 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:00.218413 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:00.325683 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:00.355302 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:00.407884 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:18:00.415796 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:00.421249 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:00.437644 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:00.441690 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:00.454032 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:00.461035 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:00.495535 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:00.530798 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:00.536221 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:00.555065 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:00.620481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:00.637548 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:00.640821 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:00.674603 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:00.675419 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:00.692456 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:18:00.701407 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:00.707399 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:00.774627 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 14:18:00.775458 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:00.795900 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:00.849848 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:00.871634 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:00.871773 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:00.872581 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:00.890045 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:00.934024 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:01.014892 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:01.063902 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:01.081149 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:01.106132 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:01.115444 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:01.117266 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:01.129153 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:01.151417 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:18:01.157214 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:01.173966 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:01.177817 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:01.177989 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:18:01.191426 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:01.209296 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:01.244333 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:01.313419 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:01.327543 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:01.372186 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(b112f8fb89c7404b87a4c6e3d72fdcfd): perf score=0.033400
W20240503 14:18:01.379408 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:01.385805 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:01.401460 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:01.420415 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:01.453733 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:01.456487 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:01.500087 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:01.517943 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:01.534755 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(b112f8fb89c7404b87a4c6e3d72fdcfd) complete. Timing: real 0.161s user 0.063s sys 0.037s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41456,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":6894,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1006,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":48763,"threads_started":2,"wal-append.queue_time_us":329}
I20240503 14:18:01.536235 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(9ae190d0f4d843c990e6fdadcac677ed): perf score=0.033389
I20240503 14:18:01.577054 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(9ae190d0f4d843c990e6fdadcac677ed) complete. Timing: real 0.041s user 0.034s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1573,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":723,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":206,"threads_started":1,"wal-append.queue_time_us":287}
I20240503 14:18:01.578104 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(b112f8fb89c7404b87a4c6e3d72fdcfd): 4838 bytes on disk
I20240503 14:18:01.578646 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(b112f8fb89c7404b87a4c6e3d72fdcfd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 14:18:01.579810 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(9ae190d0f4d843c990e6fdadcac677ed): 4583 bytes on disk
I20240503 14:18:01.580332 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(9ae190d0f4d843c990e6fdadcac677ed) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240503 14:18:01.581333 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:01.656013 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:01.659551 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:01.659891 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:01.670457 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:01.686940 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:01.701882 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:01.713989 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:01.743589 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:01.747998 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:01.752331 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:01.785669 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:01.822263 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:01.832191 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(fe0eebf47c494dad8edf06197cdf15bc): perf score=0.033365
W20240503 14:18:01.842762 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:01.856069 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:18:01.874864 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.042s user 0.031s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1336,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":871,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":245,"threads_started":1,"wal-append.queue_time_us":335}
I20240503 14:18:01.875963 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc): 4838 bytes on disk
I20240503 14:18:01.876505 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 14:18:01.915452 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:01.928561 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:01.936764 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:01.936950 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:01.938406 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:01.952692 25124 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:01.952881 25124 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:01.953876 25124 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:18:01.954095 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:18:01.954313 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:18:01.956765 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:01.957108 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:01.957250 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:01.957885 25124 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:01.958133 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:01.969296 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:01.969879 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:01.975931 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:02.146992 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:02.153120 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:02.177054 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:02.177454 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:02.177690 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:02.193185 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:02.200359 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:02.204949 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:02.337011 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:02.343499 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:02.375846 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:02.377952 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:02.422483 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:02.451133 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:02.524112 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:02.525214 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:02.545177 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:02.572748 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:02.572856 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:02.596925 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:02.597489 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:02.623170 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:02.728051 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:02.734651 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:02.795694 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:02.800959 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:02.811691 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:02.814724 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:02.835304 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:02.858148 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:02.870642 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:02.884928 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(aba6537cdb87400cb8b108b1fbfa6c0f): perf score=0.033375
W20240503 14:18:02.888295 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:02.888957 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:02.925603 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(aba6537cdb87400cb8b108b1fbfa6c0f) complete. Timing: real 0.040s user 0.033s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":471,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":675,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":602,"threads_started":2,"wal-append.queue_time_us":312}
I20240503 14:18:02.926785 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(aba6537cdb87400cb8b108b1fbfa6c0f): 4921 bytes on disk
I20240503 14:18:02.927349 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(aba6537cdb87400cb8b108b1fbfa6c0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240503 14:18:02.963765 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:02.973592 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:18:02.977209 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:03.014842 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:03.035621 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:03.045565 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:03.053895 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:03.054617 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:03.068646 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:03.075876 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:03.076752 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:03.143276 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:03.269189 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:03.291160 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:03.300745 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:03.309974 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:03.315655 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:03.338712 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:03.362735 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:03.379638 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:03.430265 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(fda4460865854505b5220d7ec3b34c57): perf score=0.033402
W20240503 14:18:03.459702 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:18:03.466850 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(fda4460865854505b5220d7ec3b34c57) complete. Timing: real 0.036s user 0.032s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":725,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1267,"threads_started":2,"wal-append.queue_time_us":297}
I20240503 14:18:03.468166 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(fda4460865854505b5220d7ec3b34c57): 4583 bytes on disk
I20240503 14:18:03.468855 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(fda4460865854505b5220d7ec3b34c57) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240503 14:18:03.469664 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(32927a23842b4725b9dfafbac47fe754): perf score=0.033363
W20240503 14:18:03.475430 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:03.532860 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(32927a23842b4725b9dfafbac47fe754) complete. Timing: real 0.063s user 0.038s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":2490,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1281,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7627,"threads_started":1,"wal-append.queue_time_us":7513}
I20240503 14:18:03.534219 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(32927a23842b4725b9dfafbac47fe754): 4923 bytes on disk
I20240503 14:18:03.534976 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(32927a23842b4725b9dfafbac47fe754) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 14:18:03.535074 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:03.542282 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:18:03.550707 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:03.555200 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:03.575320 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:03.576474 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:03.581276 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:03.583985 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:03.599083 24773 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:03.599303 24773 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:03.600004 24773 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:03.602523 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:03.603312 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:03.603435 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:03.603816 24773 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:03.604310 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:03.692062 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:03.703348 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:03.716506 24773 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:03.716713 24773 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
W20240503 14:18:03.717304 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:18:03.717386 24773 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:03.719030 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:03.719262 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:03.719377 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:03.719677 24773 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:03.778406 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:18:03.781071 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:03.791617 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:03.792778 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:03.799489 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:03.808583 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:03.818228 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:03.830322 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:03.876344 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:03.880698 2498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 44294.6us
W20240503 14:18:03.907490 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:03.940093 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:04.008620 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:04.028146 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:04.030025 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:04.038568 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:04.040999 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:04.047163 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:04.058208 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:04.109405 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:04.169651 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:04.231207 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:04.244494 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:04.255028 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:04.268630 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:04.273331 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:04.289062 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:04.315955 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:04.329658 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:04.389967 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:04.391597 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:04.396173 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:04.435971 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:04.487498 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:04.488488 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:04.535769 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:04.616106 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:04.654287 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:04.676591 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240503 14:18:04.679605 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:04.680545 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:04.683624 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:04.690531 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:18:04.691823 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:04.707674 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:04.747599 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:04.761699 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:18:04.762729 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:04.889297 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:04.912415 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:04.915853 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:04.921460 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:04.935447 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 14:18:04.941053 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:04.942767 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:04.968350 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:04.984161 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:05.004480 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:05.050513 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:05.080868 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:05.123764 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:05.127434 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:05.154858 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:18:05.168001 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:05.196847 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:05.201956 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:05.218402 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:05.219938 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:05.224025 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:05.284755 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:05.294430 24402 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:05.294667 24402 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:05.295728 24402 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:05.297174 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:05.297418 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:05.297564 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:05.297732 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 1 yes votes; 3 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:05.297984 24402 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:05.402858 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:05.405421 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:05.414505 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:05.423147 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:05.426318 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:05.429978 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:05.434190 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:18:05.436673 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:05.465591 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:05.480322 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:05.490526 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:05.517087 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:05.517675 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:05.544026 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(5f631c7713d24f4caf7550bb05147615): perf score=0.033344
W20240503 14:18:05.573100 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:05.578195 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(5f631c7713d24f4caf7550bb05147615) complete. Timing: real 0.034s user 0.022s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1502,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":536,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":463,"threads_started":2,"wal-append.queue_time_us":359}
I20240503 14:18:05.580842 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(5f631c7713d24f4caf7550bb05147615): 4583 bytes on disk
I20240503 14:18:05.581329 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(5f631c7713d24f4caf7550bb05147615) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:18:05.582100 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(b359fb8d9afc4c9c856897a722100304): perf score=0.033333
W20240503 14:18:05.607259 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:05.630759 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:05.675159 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:05.678030 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:05.678149 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:05.689247 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:05.703466 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(b359fb8d9afc4c9c856897a722100304) complete. Timing: real 0.121s user 0.061s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1472,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":841,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10025,"threads_started":1,"wal-append.queue_time_us":7587}
I20240503 14:18:05.704949 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(b359fb8d9afc4c9c856897a722100304): 4838 bytes on disk
I20240503 14:18:05.712908 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(b359fb8d9afc4c9c856897a722100304) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 14:18:05.717689 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:05.838050 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:05.884909 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:05.886925 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:05.892578 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:05.897166 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:05.919569 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:05.922204 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:05.951673 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:18:05.952541 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:05.990590 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:06.048213 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:06.054699 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:06.099886 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:06.104434 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:06.116940 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:06.132054 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:06.134881 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:06.139379 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:06.154537 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:06.189458 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:06.239730 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:06.265403 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:06.322849 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:06.334911 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:06.360679 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:06.379523 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:06.401084 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:06.402115 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:06.413884 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:06.420214 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:06.421077 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:06.424952 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:06.431043 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:06.486152 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:06.493069 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:06.509914 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:06.520426 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:18:06.552978 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:06.566682 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:06.579224 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:06.682029 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:06.727383 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:06.779392 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:06.810451 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:06.824167 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:06.833986 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:06.839602 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:06.852685 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:06.859422 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:06.860391 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:06.898697 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:06.899214 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:06.942806 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:06.982635 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:18:07.039639 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:07.039958 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:07.043241 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:07.043807 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:07.065976 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:07.208230 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:07.208391 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:07.224213 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:07.246102 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:07.248530 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:07.267935 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:07.276996 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:07.297452 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:07.297659 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:07.350410 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:07.367616 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:07.387962 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:07.411926 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:07.423902 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:07.443135 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:07.472033 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(af99824d1853471b867e4ff1f01e9367): perf score=0.033343
I20240503 14:18:07.527922 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:07.549933 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:07.551728 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:07.563733 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:07.563897 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:07.570255 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(af99824d1853471b867e4ff1f01e9367) complete. Timing: real 0.098s user 0.041s sys 0.037s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":25600,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":12618,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1003,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":38514,"threads_started":2,"wal-append.queue_time_us":13101}
I20240503 14:18:07.571780 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(af99824d1853471b867e4ff1f01e9367): 4583 bytes on disk
I20240503 14:18:07.575682 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(af99824d1853471b867e4ff1f01e9367) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240503 14:18:07.590353 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:07.590682 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:07.637980 25824 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:07.638217 25824 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:07.639469 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:18:07.639722 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
I20240503 14:18:07.640667 25824 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:07.641017 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:07.642397 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:07.642490 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:07.642735 25824 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:07.693646 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:07.693897 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:07.707918 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:07.764900 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:07.767562 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:07.773862 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:07.786693 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:07.788749 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:07.808487 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:18:07.827872 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(a9a474791f9b4585aa0114162705c96f): perf score=0.033401
W20240503 14:18:07.845207 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:07.881107 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(a9a474791f9b4585aa0114162705c96f) complete. Timing: real 0.053s user 0.032s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1350,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1447,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191,"threads_started":1,"wal-append.queue_time_us":283}
I20240503 14:18:07.882437 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(a9a474791f9b4585aa0114162705c96f): 4923 bytes on disk
I20240503 14:18:07.882988 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(a9a474791f9b4585aa0114162705c96f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240503 14:18:07.911895 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:07.924973 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:07.981324 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:07.985028 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:07.991444 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:07.992287 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:08.015152 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:08.042865 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:08.075436 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:08.076994 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:08.175122 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:08.186046 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:08.189430 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:08.223520 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:08.232190 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:08.238138 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:08.243065 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:08.246580 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:08.252807 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:08.317731 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:08.415935 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:08.448204 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:08.449304 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:08.451906 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:08.455399 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:08.456319 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:08.459548 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:08.462023 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:08.475612 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:08.671231 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:08.673913 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:08.746026 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:08.746933 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:18:08.747304 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:08.758137 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:08.772946 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:08.788049 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:08.805791 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:08.841979 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:08.862090 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:08.868393 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:08.877899 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:08.883208 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:08.912833 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:08.939935 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:08.954478 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:08.954774 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:08.981645 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:08.995872 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:09.002378 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:09.021541 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:09.052521 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:09.075431 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:09.091324 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:09.097640 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:09.121104 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:09.121739 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:09.124588 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:09.178676 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:09.219820 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:09.296833 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:09.302229 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:09.326108 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:09.343148 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:09.373562 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:09.377193 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:09.390096 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:09.425940 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:09.455989 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:09.459240 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:18:09.567168 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:09.575027 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:09.580515 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:09.593742 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:09.594113 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:09.655395 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:09.693454 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:18:09.694576 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:09.705861 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:18:09.721535 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:09.772888 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:09.797881 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 133 similar messages]
I20240503 14:18:09.799315 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:09.806144 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:09.813392 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:09.832679 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:09.833990 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:09.849624 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:09.887735 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:09.904683 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:09.978276 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:18:10.008169 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:10.009564 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:10.014739 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:10.023380 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:10.024672 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:10.044081 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:10.044386 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:10.056586 25824 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:10.056767 25824 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:10.057551 25824 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:18:10.059299 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:18:10.059531 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:18:10.060449 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:10.060722 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:10.060806 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:10.061281 25824 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:10.106519 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:10.113808 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:10.158967 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:18:10.171126 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:10.253760 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:18:10.283617 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:10.295068 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:10.295478 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:10.315836 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:10.329499 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:10.339499 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:10.352002 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:10.427312 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:10.489625 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:10.535622 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:10.549376 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:10.555733 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:10.587868 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:10.603562 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:10.609716 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:10.617300 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:10.688422 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:10.711071 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:10.737100 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:10.752426 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:10.772214 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:10.791399 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:10.835371 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:10.855401 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:10.875991 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:10.904740 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:10.950457 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:10.994215 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 14:18:11.006708 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:11.009065 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:11.026243 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:11.033061 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:11.042503 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:11.043785 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:11.072678 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:11.153343 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:11.175163 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:11.179929 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:11.217809 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:11.220992 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:11.221273 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:11.222405 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:11.225744 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:11.230109 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:11.246412 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:11.273583 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:11.350875 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:11.360025 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:11.369007 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:11.369093 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:18:11.435863 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:11.436841 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:11.450616 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:11.464656 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:11.470597 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:11.471022 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:11.481300 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:11.482770 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:11.525130 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:11.604980 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:11.694824 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:11.713896 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:11.714421 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:11.715483 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:11.716235 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:11.731489 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:11.736153 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:11.750337 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:11.824410 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:11.847965 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:11.902310 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:11.930471 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:11.938203 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:11.939468 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:11.947590 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:11.958833 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:11.990468 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:11.992908 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:12.035367 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:12.083742 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:12.146865 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:12.151502 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:12.158439 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:12.166020 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:12.166851 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:12.175618 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:12.199316 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:12.199453 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:12.274081 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:12.346029 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:12.383437 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:12.394961 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:12.398074 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:12.406708 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:12.410414 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:12.431577 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:12.502264 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:12.505417 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:12.532081 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:12.537009 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:12.608536 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:12.621850 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:18:12.626785 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:12.629812 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:12.650491 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:12.683229 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:12.683645 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:12.720831 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:12.803555 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:12.876257 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:12.920186 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:12.922513 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:12.971632 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:12.976742 25840 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:12.976945 25840 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:12.977957 25840 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:18:12.978641 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:18:12.978919 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:18:12.979895 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:12.980042 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:12.980178 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:12.980465 25840 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:13.043738 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:13.055215 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:13.056664 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:13.057631 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:13.064864 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:13.080089 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:13.080719 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:13.088061 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:13.094105 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:13.111492 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:13.264608 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:13.284487 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:13.286170 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:13.288316 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:13.292444 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:13.303442 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:13.331346 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:13.338497 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:13.341276 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:13.372802 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:13.412566 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:13.441210 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:13.442168 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:13.503973 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:13.507553 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:13.525532 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:13.528329 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:13.533641 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:13.573437 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:13.582610 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:13.650475 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:13.662492 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:13.702844 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:13.718559 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:13.727201 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:13.748359 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:18:13.758800 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:13.768112 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:13.785418 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:13.785435 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:13.788242 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:13.817658 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:13.834333 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:13.839709 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:13.908377 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:18:13.915623 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:13.935725 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:13.996791 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:14.007674 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:14.009364 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:14.010228 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:14.016745 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:14.039139 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:14.050546 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:14.067348 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:14.194396 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:14.196688 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:14.206478 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:14.208097 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:14.221792 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:14.226209 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:14.322324 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:14.348588 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:14.356314 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:14.424501 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:14.445092 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:14.453047 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:14.455490 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:14.457638 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:14.461891 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:14.484282 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:14.537568 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:14.568982 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:14.576886 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:14.591665 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:14.599098 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:14.616250 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:14.662818 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:14.684950 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:14.689468 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:14.715080 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:14.721163 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:14.725546 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:14.730142 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:14.769269 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:18:14.799914 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:14.878764 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:18:14.885639 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240503 14:18:14.887274 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:14.959939 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:14.989051 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:18:15.072292 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:15.076939 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:18:15.080197 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:15.105568 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:15.122285 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:15.137269 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:18:15.143435 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:15.147962 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:15.151090 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:15.166162 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:15.166808 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:15.189918 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:18:15.376629 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:15.378136 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:15.391526 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:15.403937 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:15.421736 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:15.434405 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:15.442221 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:15.443493 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:15.490535 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:15.531677 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:15.557258 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:15.568734 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:15.619436 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:15.627606 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:15.627607 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:15.636124 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:15.646462 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:15.654762 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:15.667631 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:15.667644 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:15.826122 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:15.884685 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:15.900696 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:15.915577 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:15.926028 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:15.949272 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:15.969410 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:15.983513 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:16.014261 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:16.023587 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:16.035534 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:16.078152 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:16.080119 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:16.090943 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:18:16.092686 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:16.119249 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:16.127707 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:16.156157 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:16.160244 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:16.161659 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:16.180425 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:16.192878 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:16.211622 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:16.246114 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:16.289523 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:16.298837 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:16.312817 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:16.366961 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:16.368836 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:16.387490 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:16.387538 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:16.389215 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:16.401335 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:16.402765 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:16.412083 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:16.412578 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:16.528803 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:16.539614 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:16.562716 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:16.574927 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:18:16.606916 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:16.615278 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:16.617909 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:16.620082 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:16.637089 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:16.775125 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:16.803174 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:16.822001 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:16.837208 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:16.854058 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:16.858882 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:16.864256 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:16.891041 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:16.909346 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:16.917435 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:16.947666 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:17.018085 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:17.034699 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:17.098673 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:17.115824 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:17.215682 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:17.224088 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:17.253847 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:17.277452 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:17.283970 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:17.284483 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:17.285183 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:17.285660 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:17.302314 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:17.310108 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:17.311244 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:17.343356 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:17.395082 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:17.484826 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:17.486356 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:17.491099 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:17.517174 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:17.524430 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:17.528954 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:17.536417 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:17.581336 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:17.708585 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:17.752971 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:17.755865 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:17.771880 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:17.773819 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:18:17.778198 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:17.778326 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:17.782088 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:17.791206 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:17.819164 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:17.832331 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:17.995292 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:18.007313 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:18.017551 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:18.018885 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:18.028955 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:18.046198 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:18.125463 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:18.137856 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:18.138119 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:18.195925 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:18.209810 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:18.214748 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:18:18.218827 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:18.231412 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:18.239703 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:18.250061 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:18.277585 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:18.286199 26682 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:18.286388 26682 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:18.287048 26682 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:18.289255 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:18.300240 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:18.300766 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:18.301097 26682 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:18.326970 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:18.372835 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:18.469642 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:18.492828 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:18.501374 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:18.505049 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:18.525691 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:18.528357 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:18.538527 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:18.574162 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:18.581701 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:18.629823 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:18.637403 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:18.638898 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:18.643018 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:18.665349 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:18.673941 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:18.677724 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:18.680112 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:18.712661 27542 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:18.712859 27542 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:18.713657 27542 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:18.716177 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:18.716445 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:18.716584 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:18.716995 27542 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:18.718351 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:18.729172 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:18.737589 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:18.746186 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:18.756249 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:18.772452 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:18.834869 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:18.838466 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:18.909495 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:18.952544 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:18.959539 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:18.973109 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:18.976337 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:19.001399 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:19.024456 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:19.040796 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:19.083451 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:19.139036 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:19.149204 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:19.159646 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:19.192729 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:19.247735 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:19.361466 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:19.377110 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:18:19.386713 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:19.400216 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:19.410140 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:19.410197 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:19.412046 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:19.418191 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:19.430698 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:19.445547 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:19.467608 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:19.547963 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:19.553252 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:19.602705 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:19.605471 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:19.616750 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:19.650167 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:19.651363 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:19.671326 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:19.699290 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:19.729009 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:19.765267 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:19.773546 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:19.792019 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:18:19.792935 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:19.885310 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:19.904196 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:19.927047 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:19.927264 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:19.931109 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:19.943846 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:18:19.948225 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:18:19.952591 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:20.014555 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:20.043107 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:18:20.070068 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:20.076861 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:20.113864 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:20.114969 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:20.122145 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:20.129323 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:20.139402 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:20.146167 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:20.151582 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:20.164067 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:20.350457 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:20.364238 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:18:20.388525 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:20.405723 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:20.407945 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:20.412024 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:20.414606 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:20.432065 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:20.434185 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:20.441790 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:20.512604 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:20.597162 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:20.603411 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:20.603753 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:20.604990 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:20.608029 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:20.685775 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:20.695963 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:20.771967 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:20.795202 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:20.813800 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:20.816334 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:20.821898 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:20.823558 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:20.828514 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:20.844477 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:20.855504 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:20.893342 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:20.960572 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:20.974152 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:20.999435 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:21.032243 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:21.042076 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:21.060465 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:21.060998 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:21.066941 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:21.073832 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:21.117637 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:21.161202 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:21.161247 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:21.183286 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:21.230747 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:21.230890 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:21.257084 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:18:21.257923 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:21.269156 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:21.290623 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:21.361083 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:21.375044 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:18:21.442543 28018 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:21.442737 28018 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:21.443622 28018 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:21.449617 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:21.449800 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:21.449913 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:21.450345 28018 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:21.461831 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:21.468746 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:21.471354 28057 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:21.471521 28057 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:21.472364 28057 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:21.474197 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:21.475837 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:21.475935 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:21.476374 28057 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:21.483595 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:21.494808 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:21.498591 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:21.518741 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:21.522264 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:21.549907 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:21.573002 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:21.593533 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:21.650519 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:21.697540 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:21.714705 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:21.715770 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:21.722483 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:21.725320 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:21.744343 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:21.750507 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:21.782439 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:21.799886 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:21.811894 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:21.898156 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:21.934653 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:21.952497 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:21.971457 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:21.972245 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:21.979890 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:21.984215 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:21.994422 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:22.023726 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:18:22.053825 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:22.054355 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:22.070036 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:22.088214 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:22.088886 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:22.129526 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:22.171972 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:22.188211 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:22.202412 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:22.213733 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:22.223646 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:22.224462 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:22.281191 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:22.328753 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:22.389113 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:22.411942 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:22.423118 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:22.432072 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:22.436864 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:22.456713 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:22.496467 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:22.527696 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:22.594445 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:22.606561 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:22.614331 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:22.623018 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:22.630674 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:22.640007 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:22.643463 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:22.659201 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:22.728094 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:22.859939 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:22.861605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:22.861785 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:22.874666 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:22.882586 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:22.885713 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:22.887332 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:22.904145 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:22.948200 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:23.032989 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:23.088711 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:23.094532 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:23.095991 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:23.106606 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:23.118592 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:23.126556 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:23.132690 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:23.153386 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:18:23.231827 28057 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:23.232017 28057 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:23.232673 28057 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:23.234185 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:23.234375 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:23.234488 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:23.234750 28057 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:23.351430 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:23.370564 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:23.371909 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:23.389006 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:23.440022 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:23.451675 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:23.489116 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:23.505545 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:23.532120 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:23.532827 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:23.551625 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:23.568353 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:23.573685 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:23.587724 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:23.592569 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:23.596628 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:23.605707 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:23.619419 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:23.738823 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:23.752784 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:23.754418 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:23.768803 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:23.787376 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:23.793089 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:23.822496 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:23.840759 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:23.845854 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:23.846418 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:23.884187 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:23.939471 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:23.950929 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:23.979244 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:23.996802 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:24.013337 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:24.014777 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:24.018555 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:24.045651 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:24.082796 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:24.214458 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:24.218458 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:24.231434 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:24.231623 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:24.242960 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:24.254477 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:24.295946 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:24.309825 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:24.311067 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:24.362273 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:24.363906 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:24.365227 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:24.378613 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:24.439096 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:24.440863 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:24.445782 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:24.446174 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:24.446961 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:24.467166 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:24.467494 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:24.472867 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:24.498814 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:24.523564 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:24.573657 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:24.588201 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:18:24.679867 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:24.680931 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:24.693884 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:24.699352 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:24.703456 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:24.724154 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:18:24.793852 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:24.800998 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:18:24.845168 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:24.881435 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:24.897415 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:24.909197 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:24.909399 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:24.928089 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:24.934540 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:24.935384 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:24.936204 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:24.955896 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:18:24.962989 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 14:18:24.963819 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:24.980901 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:24.995699 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:25.005884 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:25.109563 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:25.113466 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:25.127692 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:25.132999 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:25.137270 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:18:25.138959 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:25.235148 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:25.293933 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:25.323531 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:25.353890 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:25.389662 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:25.509881 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:25.512775 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:25.531888 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:25.547529 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:25.556099 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:25.562696 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:25.581076 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:25.606979 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:25.639714 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:25.753754 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:25.771988 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:25.781002 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:25.785099 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:25.790059 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:25.823864 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:25.828023 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:25.889575 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:25.905301 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:25.910591 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:25.920760 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:25.938508 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:25.991082 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:25.992187 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:25.993013 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:26.013942 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:26.016273 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:26.016909 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:26.019924 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:26.029170 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:26.047964 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:26.093314 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:26.126616 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:26.141299 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:26.239487 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:26.245924 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:26.246300 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:26.254745 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:26.264395 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:26.270027 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:26.274976 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:26.307399 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:18:26.355058 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:26.362471 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:26.369858 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:26.381726 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:26.407217 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:26.408540 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:26.458400 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:26.488416 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:26.514155 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:26.525986 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:26.530741 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:26.544162 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:26.658159 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:26.701164 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:26.731845 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:26.745611 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:26.759569 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:26.763494 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:26.764051 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:26.766574 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:26.769357 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:26.772794 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:26.838570 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:26.918035 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:26.924368 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:26.946571 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:26.964308 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:26.965858 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:26.969029 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:26.973563 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:26.992717 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:27.004976 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:18:27.009764 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:27.021881 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:27.028205 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:27.039496 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:27.047590 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:27.082886 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:27.122351 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:27.174144 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:27.175657 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:27.183243 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:27.194994 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:27.196228 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:27.229503 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:27.265102 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:27.294378 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:27.377748 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:27.415879 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:27.449829 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:27.474244 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:27.491312 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:27.522631 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:27.595155 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:27.615722 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:27.618455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:27.634644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:27.636073 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:27.648730 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:27.672397 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:27.786134 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:27.823541 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:27.833757 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:27.839066 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:27.840826 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:27.857443 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:27.971973 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:28.001394 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:28.023586 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:28.065529 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:28.071085 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:28.079414 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:28.084673 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:28.091259 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:28.092886 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:28.100682 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:18:28.106599 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:28.190382 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:28.202394 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:28.223488 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:28.292007 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:28.305362 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:28.308871 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:28.325982 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:28.335434 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:28.359892 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:28.363560 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:28.363855 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:28.387375 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:28.401803 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:28.404860 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:28.436411 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:28.495229 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:28.559005 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:28.560818 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:28.575453 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:28.584022 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:28.590204 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:28.608530 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:28.654728 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:28.701368 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:28.727777 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:28.764307 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:28.771553 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:28.774644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:28.778560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:28.781633 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:28.783423 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:28.789005 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:28.901445 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:28.915231 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:28.916086 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:28.947119 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:28.950455 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:28.956943 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:28.978000 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:28.981110 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:28.990963 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:29.019516 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:29.026372 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:29.132076 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:29.183715 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:29.199347 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:29.207360 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:29.222986 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:29.226969 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:29.236886 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:29.291399 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:29.312914 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:29.326169 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:29.348146 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:29.379397 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:29.399644 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:29.484155 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:29.511977 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:29.526822 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:29.557503 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:29.571558 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:29.577973 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:29.593019 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:29.593881 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:29.629911 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:18:29.630872 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:29.645030 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:29.651757 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:29.662267 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:29.663486 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:29.667538 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:29.832613 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:18:29.834791 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:29.843142 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:29.852207 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:29.879649 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:29.881091 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:29.884857 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:29.907999 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:29.945981 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:29.973304 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:29.987600 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:29.994550 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:29.999320 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:18:30.011931 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 14:18:30.017709 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:30.046180 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:30.090636 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:30.100422 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:30.131851 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:30.137887 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:30.167604 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:30.204632 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:30.229295 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:30.258637 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:18:30.367018 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:30.373056 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:30.385454 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:30.386248 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:30.398315 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:30.403743 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:30.420297 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:30.464953 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:30.489537 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:30.606854 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:30.626329 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:30.633101 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:30.653601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:30.658120 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:30.664630 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:30.681494 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:30.687289 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:30.691589 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:30.695066 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:18:30.695209 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:30.736753 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:30.774711 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:30.790565 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:30.814013 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:30.869722 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:30.872000 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:30.882360 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:30.905469 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:30.920169 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:30.926488 29469 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:30.926666 29469 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:30.947235 29469 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:30.948129 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:30.950697 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:30.954234 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:18:30.954475 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:18:30.959540 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:30.959681 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:30.959945 28967 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:31.004947 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:31.038081 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:31.116263 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:31.121279 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:31.136081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:31.139418 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:31.154788 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:31.168097 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:31.178782 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:31.187044 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:31.228121 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:31.240339 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:31.263226 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:31.272186 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:31.314883 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:31.335290 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:31.348745 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:31.365938 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:31.378172 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:31.395345 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:31.424865 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:31.460705 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:31.465909 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:18:31.607944 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:31.621011 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:31.643597 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:31.696344 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:31.787864 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:31.800113 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:31.805302 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:31.813735 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:31.827922 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:31.830265 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:31.837811 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:31.839166 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:31.840699 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:31.869810 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:31.947799 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:31.948968 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:31.959900 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:32.007544 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:32.011967 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:32.020303 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:32.021322 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:32.030186 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:32.050788 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:32.053601 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:32.094807 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:32.129037 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:32.155606 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:32.159300 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:32.229574 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:32.248273 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:32.256191 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:32.265733 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:32.275811 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:32.394832 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:32.399602 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:32.409564 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:32.410950 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:32.443743 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:32.475793 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:32.480517 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:32.487771 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:32.492184 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:32.519389 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:18:32.529510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:32.540132 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:32.555841 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:32.558264 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:32.590909 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:32.717013 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:32.717087 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:32.722270 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:32.724179 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:32.733647 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:32.741930 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:32.770027 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:32.831212 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:32.883100 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:32.908589 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:32.925503 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:32.930663 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:32.945402 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:32.945976 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:32.947968 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:32.948995 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:33.028767 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:33.097319 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:33.144137 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:33.146772 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:33.160116 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:33.168289 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:33.181236 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:33.181982 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:33.189132 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:33.296146 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:33.325035 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:18:33.388196 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:33.396773 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:33.397439 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:33.403272 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:33.407423 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:33.407681 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:33.407958 29896 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:33.408128 29896 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:33.409637 29896 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:33.410738 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:33.412838 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:33.412962 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:33.413421 29896 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:33.428418 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:33.428494 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:33.429208 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:33.457587 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:33.523921 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:33.524657 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:33.536217 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:33.634522 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:33.647239 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:33.669436 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:33.778364 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:33.793092 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:33.793852 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:33.837899 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:33.839819 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:33.840773 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:33.856243 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:33.858958 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:33.862938 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:33.884060 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:34.017371 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:34.027798 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:34.048764 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:34.049175 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:34.053798 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:34.066265 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:34.068509 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:34.093190 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:34.094811 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:34.103420 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:34.263365 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:34.275465 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:34.276284 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:34.281730 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:34.282100 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:34.282557 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:34.303478 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:34.376184 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:34.378749 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:34.452689 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:34.472970 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:34.473453 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:34.487607 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:34.501619 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:34.536634 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:34.537636 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:34.540201 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:34.547415 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:34.567560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:34.595705 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:34.651432 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:34.691085 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:34.749468 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:34.751878 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:34.754557 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:34.761356 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:34.766273 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:34.782043 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:34.793183 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:34.814707 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:34.832533 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:34.869673 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:18:34.871670 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:34.872138 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:34.915259 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:34.958101 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:34.967082 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:34.975986 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:35.000581 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:35.010602 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:35.016391 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:18:35.067323 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:18:35.067867 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:35.084213 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:35.103605 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:18:35.108361 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:35.110117 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:35.136301 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:35.164877 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:35.174360 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:35.174533 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:35.193836 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:35.194485 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:35.199553 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:35.233505 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:35.283043 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:18:35.329169 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:35.333683 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:35.434177 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:35.451128 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:35.464252 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:35.470995 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:35.483439 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:35.484396 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:35.485318 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:35.526502 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:35.550025 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:35.652458 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:35.679606 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:35.731961 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:35.764122 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:35.877157 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:35.877537 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:35.882964 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:35.886461 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:35.898528 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:35.903626 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:18:35.904636 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:35.911077 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:35.963716 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:35.978402 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:36.031910 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:36.083792 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:36.088878 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:36.090148 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:36.103657 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:36.112121 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:36.113427 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:36.175119 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:36.192961 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:36.231735 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:36.267840 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:36.294652 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:36.297041 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:18:36.309057 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:36.312213 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:36.312893 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:36.314821 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:36.329612 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:36.370293 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:36.426749 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:36.441566 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:36.470739 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:36.529856 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:36.540258 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:36.541532 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:36.558046 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:36.563767 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:36.573072 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:36.583774 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:36.587718 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:18:36.604840 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:36.667158 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:36.710060 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:36.773003 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:36.791791 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:36.795614 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:36.832031 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:36.833561 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:36.834251 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:36.884657 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:36.939821 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:36.952124 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:37.006675 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:37.009025 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:18:37.020975 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:37.027760 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:37.040625 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:37.049774 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:37.050797 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:37.051972 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:37.092679 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:37.174844 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:37.183907 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:37.204332 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:37.256564 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:37.278779 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:37.284484 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:37.298116 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:37.302238 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:37.307408 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:37.309801 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:37.381213 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:37.396179 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:37.407792 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:37.462357 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:37.473698 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:37.490922 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:37.491217 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:37.497543 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:37.499007 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:37.529714 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:37.536615 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:37.536849 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:37.565466 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:37.591408 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:18:37.679174 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:37.724135 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(32927a23842b4725b9dfafbac47fe754): perf score=0.033337
I20240503 14:18:37.755765 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(32927a23842b4725b9dfafbac47fe754) complete. Timing: real 0.031s user 0.023s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":335,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":845,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":468,"threads_started":2,"wal-append.queue_time_us":345}
I20240503 14:18:37.756862 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(32927a23842b4725b9dfafbac47fe754): 4923 bytes on disk
I20240503 14:18:37.757395 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(32927a23842b4725b9dfafbac47fe754) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240503 14:18:37.767410 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:37.781522 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:37.785665 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:37.799780 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:37.865379 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:37.902030 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:37.932914 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:37.946314 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:37.951942 30481 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:37.952164 30481 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:37.953081 30481 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:37.954972 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:37.956403 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:37.956548 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:37.956854 30481 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:37.968350 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:37.976262 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:37.977622 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:37.981577 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:38.031576 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:38.057343 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:38.127744 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:38.150655 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:38.153350 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:38.172864 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:38.176951 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:38.185052 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:38.233161 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:38.245563 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:38.298048 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:38.319983 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:38.347586 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:38.367322 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:18:38.370147 28018 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:38.370340 28018 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:38.371181 28018 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:38.373612 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:38.374337 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:38.374439 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:38.374696 28018 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:38.375790 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:38.402181 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:38.412951 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:38.427423 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:38.428772 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:38.440964 30481 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:38.441426 30481 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:38.442920 30481 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:18:38.458070 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:38.461771 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:38.472173 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:38.472493 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:38.472894 30481 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:38.553663 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:38.578269 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:38.581418 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:38.584424 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:38.618955 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:38.642258 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:38.645771 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:38.648257 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:38.664261 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:38.669876 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:38.681663 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:38.686239 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:38.767727 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:38.785004 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:38.804140 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:38.807755 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:38.830451 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:38.846221 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:38.871503 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:38.886682 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:38.888413 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:38.938271 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:38.948716 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:38.956238 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:38.969043 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:38.986959 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:39.031049 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:39.112676 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:39.125770 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:39.127002 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:39.151010 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:39.160759 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:39.162278 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:39.190580 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:39.297516 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:39.323083 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:39.324190 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:39.390399 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:39.412997 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:39.418121 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:39.419283 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:39.421984 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:39.441136 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:39.466161 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:39.479802 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:39.526556 30678 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:39.526764 30678 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:39.527498 30678 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:39.528753 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:39.528868 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:39.528949 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:39.529184 30678 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:39.534853 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:39.572564 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:39.578385 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:39.590365 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:39.610101 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:39.616781 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:39.619632 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:39.637094 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:39.638051 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:39.743358 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:39.757882 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:39.783569 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:39.823683 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:39.841482 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:39.903771 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:39.906015 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240503 14:18:39.906787 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:39.920204 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:40.024572 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240503 14:18:40.037000 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:40.057406 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:40.060072 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:40.060631 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:40.063956 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:40.069109 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:40.095058 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:40.102427 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:40.135713 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:18:40.142297 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:40.162586 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:40.185182 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:18:40.248692 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:40.259265 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:40.260320 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:40.261477 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:40.261843 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:40.268618 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:40.284677 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:40.340663 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:40.372821 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:40.379340 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:18:40.380645 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:40.405100 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:40.411597 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:40.422302 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:40.463704 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:40.479404 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:40.485405 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:40.485755 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:40.521286 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:40.568814 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:40.639444 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:40.640645 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:40.722505 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:40.727337 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:40.728217 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:40.730583 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:40.732327 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:40.738845 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:40.747961 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:40.769280 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:40.888520 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:40.937139 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:40.954187 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:40.955721 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:40.958377 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:40.961849 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:40.965864 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:40.976882 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:41.009363 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:41.025135 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:41.031306 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:41.034830 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:41.050918 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:41.125924 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:18:41.155936 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:41.165844 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:41.167037 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:41.173085 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:41.191592 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:41.316911 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:41.325829 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:41.344558 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:41.370703 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:41.373980 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:41.399837 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:41.422151 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:41.427487 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:41.439330 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:41.440104 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:41.442390 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:41.442803 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:41.481616 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:41.490234 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:41.499915 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:41.521066 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:41.606185 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:41.624781 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240503 14:18:41.676514 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:41.684212 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:41.687464 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:41.709264 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:41.710492 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:41.770327 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:41.790969 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:41.877694 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:41.935113 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:41.961560 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:41.964630 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:41.992264 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:42.049548 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:42.092056 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:42.098917 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:42.109364 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:42.114341 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:42.116710 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:42.125461 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:42.128428 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:42.224663 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:42.225301 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:42.227552 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:42.234007 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:42.302073 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:42.303184 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:42.333380 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:42.336174 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:42.338228 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:42.376314 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:42.418887 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:42.519999 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:42.550868 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:42.558308 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:42.559060 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:42.569684 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:42.572232 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:42.573177 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:42.582422 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:42.586196 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:42.601022 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:42.644022 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:42.668231 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:42.713882 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:42.732117 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:42.773255 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:42.780519 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:42.786482 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:42.795446 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:42.800470 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:42.812875 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:42.826261 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:42.889809 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:18:42.976737 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:42.995824 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:43.001953 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:43.008193 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:43.011972 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:43.021275 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:43.022236 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:43.030596 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:43.052958 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:43.082873 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:43.112316 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:43.160797 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:43.219553 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:43.235297 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:43.235442 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:43.237643 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:43.254844 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:43.264835 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:43.265336 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:43.317888 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:43.396409 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:43.423149 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:43.436367 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:43.458627 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:43.471742 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:43.473556 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:43.506354 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:43.512849 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:43.526327 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:43.533989 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling FlushMRSOp(f139d77b0f0c47809e6408b2771adb73): perf score=0.033344
W20240503 14:18:43.542745 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:18:43.576963 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: FlushMRSOp(f139d77b0f0c47809e6408b2771adb73) complete. Timing: real 0.043s user 0.028s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":909,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1236,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":475,"threads_started":2,"wal-append.queue_time_us":341}
I20240503 14:18:43.578372 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(f139d77b0f0c47809e6408b2771adb73): 4838 bytes on disk
I20240503 14:18:43.579033 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(f139d77b0f0c47809e6408b2771adb73) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 14:18:43.679672 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:43.702235 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:43.713073 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:18:43.726994 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:43.729952 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:43.730974 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:43.746439 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:43.756456 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:43.764626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:43.770593 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:43.798151 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:43.811290 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:43.821941 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:43.826468 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:43.881990 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:44.009063 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:44.069039 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:44.125178 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:44.131124 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:44.151677 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:44.153484 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:44.158825 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:44.170140 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:44.175508 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:44.175949 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:44.191072 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:44.259950 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:44.285138 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:44.329644 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:44.346688 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:44.402127 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:44.405061 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:44.408325 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:44.410490 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:44.428088 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:44.485492 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:44.522284 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:44.569855 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:18:44.614171 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:44.614679 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:44.618466 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:44.623313 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:44.649726 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:44.656958 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:44.663286 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:44.683530 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:44.689780 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:44.713138 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:44.715307 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:44.736752 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:44.779773 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:44.849443 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:44.860895 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:44.878928 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:44.881342 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:44.894932 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:18:44.896874 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:44.914167 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:18:44.948884 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:45.026237 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 14:18:45.060310 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:45.080138 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:45.090729 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:45.090732 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:45.110134 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:45.120314 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:45.125156 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:45.125432 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:45.132047 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:45.144510 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:45.161036 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:45.193446 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:45.209915 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:45.242869 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:18:45.243800 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:45.267010 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:45.327553 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:45.331840 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:45.334048 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:45.349686 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:45.355211 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:45.395938 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:45.548159 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:18:45.551592 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:45.551687 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:45.572161 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:45.572731 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:45.576566 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:45.585721 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:45.608142 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:45.614516 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:45.621600 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:45.660746 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:45.698637 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:18:45.699651 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:45.801683 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:45.811143 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:45.826136 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:45.843101 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:45.848002 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:18:45.848053 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:45.856948 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:45.874785 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:45.884624 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:45.920347 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:46.087827 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:46.092586 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:46.170466 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:46.193325 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:46.235927 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:46.277033 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:46.283561 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:46.286604 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:46.296277 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:46.303746 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:46.304901 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:46.310045 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:46.318450 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:46.345644 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:46.370781 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:46.422087 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:46.436457 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:46.502588 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:46.509467 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:46.538322 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:46.545991 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:46.547052 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:46.550868 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:46.552150 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:46.567104 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:46.642663 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:46.671371 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:18:46.673028 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:46.675905 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:46.694326 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:46.761601 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:46.766420 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:46.778288 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:46.784348 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:46.793572 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:46.800688 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:46.846740 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:46.874087 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:46.877094 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:46.958106 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:46.970795 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:46.984752 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:46.987864 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:46.990796 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:47.003139 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:47.016676 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:47.069653 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:47.081895 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:47.088172 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:47.190971 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:47.192556 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:47.211761 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:47.214277 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:47.215133 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:47.216877 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:47.231626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:47.245947 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:47.279917 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:47.290890 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:47.293450 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:47.396663 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:47.408263 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:47.421289 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:47.421438 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:47.439069 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:47.440253 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:47.445191 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:47.491883 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:47.505000 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:47.538465 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:47.573830 31941 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:47.574028 31941 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:47.574932 31941 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:18:47.575218 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:18:47.575464 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:18:47.577062 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:47.577929 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:47.578030 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:47.578276 31941 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:47.623040 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:47.628549 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:47.659323 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:18:47.668262 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:47.691819 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:47.716812 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:47.759426 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:47.761291 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:47.772523 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:47.791581 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:47.799266 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling FlushMRSOp(8bf4675882e64f35bc3dc9ee06ae366d): perf score=0.033397
W20240503 14:18:47.829947 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:47.862051 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: FlushMRSOp(8bf4675882e64f35bc3dc9ee06ae366d) complete. Timing: real 0.063s user 0.055s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":347,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":786,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":530,"threads_started":2,"wal-append.queue_time_us":397}
I20240503 14:18:47.863503 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(8bf4675882e64f35bc3dc9ee06ae366d): 4923 bytes on disk
I20240503 14:18:47.864176 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(8bf4675882e64f35bc3dc9ee06ae366d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240503 14:18:47.889838 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:47.975472 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:47.983371 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:47.986768 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:47.993949 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:48.009179 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:48.012009 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:18:48.014663 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:48.018224 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:48.035058 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:48.131305 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:48.138942 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:48.179955 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:48.217912 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:48.251194 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:48.417297 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:48.421787 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:48.432994 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:48.437824 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:48.446955 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:48.462281 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:48.483562 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:48.494087 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:48.499873 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:48.544416 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:48.571780 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:48.640084 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:48.663420 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:48.663635 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:48.666585 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:48.673555 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:48.679486 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:48.683449 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:48.734733 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:48.744884 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:48.745640 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:48.859129 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:48.871426 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:48.879487 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:48.882290 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:48.888024 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:48.901957 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:48.908344 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:48.925182 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:48.974643 31943 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:48.974807 31943 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:48.975665 31943 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:48.977149 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:48.977358 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:48.977447 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:48.977660 31943 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:48.984455 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:48.985878 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:49.004070 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:49.035218 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:49.077883 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:49.077908 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:49.085896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:49.093588 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:49.115072 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:49.133011 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:49.148317 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:49.155308 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:49.210937 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:49.265703 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:49.291755 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:49.303112 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:49.303695 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:49.303900 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:49.304865 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:49.332208 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:49.347589 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:49.395041 31943 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:18:49.395249 31943 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:49.395869 31943 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:49.397416 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:49.398054 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:49.398188 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:18:49.398419 31943 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:49.411448 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:49.421167 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:49.479162 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:49.500878 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:49.519414 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:49.521510 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:49.533614 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:49.536084 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:49.541492 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:49.571831 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:49.658742 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:49.660655 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:49.715286 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:49.722575 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:49.736093 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:49.753401 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:18:49.753957 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:49.786015 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:49.801153 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:49.802590 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:49.831511 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:49.855808 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:49.859210 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:49.916908 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:18:49.977682 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:49.980407 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:49.991932 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:49.994367 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:49.996287 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:49.999013 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:50.028179 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:50.029235 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:18:50.070137 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:50.072046 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:50.098629 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:50.178575 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:50.179525 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:50.188647 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:50.201088 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:50.311764 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:18:50.374037 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:18:50.406273 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:50.409457 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:50.415055 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:50.428532 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:50.437908 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:50.445887 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:18:50.518019 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:50.549255 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:18:50.550072 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:50.579547 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:50.602453 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:50.615756 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:50.618402 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:50.632336 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:50.646337 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:50.650245 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:50.658614 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:50.658998 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:50.693284 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:50.717905 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:50.769016 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:18:50.804267 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:50.881161 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:50.889004 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:50.897611 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:50.903977 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:50.904716 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:50.907155 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:50.953594 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:51.003476 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:51.081406 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:51.087936 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:51.095674 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:51.104243 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:51.105298 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:51.110688 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:51.208039 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:51.220147 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:51.227568 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:51.234589 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:51.240271 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:51.292421 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:51.296589 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:51.306728 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:51.317230 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:51.320291 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:51.373409 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:51.394134 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:51.417363 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:51.471428 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:51.534162 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:51.540450 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:51.542196 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:51.548525 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:51.557922 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:51.560575 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:51.562253 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:51.580935 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:51.596168 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:51.606742 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:51.639556 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:51.646041 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:51.735141 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:51.754392 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:51.759969 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:18:51.763126 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:51.777319 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:51.777436 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:51.803608 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:51.806824 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:51.812075 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:51.819489 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:51.840147 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:51.870004 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:51.948446 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:51.954506 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:52.019312 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:52.031424 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:52.035339 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:52.036315 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:52.051590 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:52.051640 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:52.101861 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:52.123749 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:52.322484 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:52.333076 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:52.370608 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:52.390014 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:52.417726 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:52.440033 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:52.449347 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:52.457580 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:52.473453 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:52.489145 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:52.497293 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:52.528674 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:52.574903 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:52.664539 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:52.671327 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:52.675981 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:18:52.677678 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:52.680716 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:52.693284 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:18:52.694723 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:52.695672 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:52.722601 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:52.743564 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:52.830745 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:52.899902 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:52.901937 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:52.910704 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:52.916486 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:52.919255 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:52.985183 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:52.994975 32699 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:52.995153 32699 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:52.995900 32699 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:52.997431 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:52.997553 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:52.997646 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:52.997859 32699 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:53.000424 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:53.021180 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:53.032734 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:53.073658 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:53.076970 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:53.083415 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:53.097292 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:53.113620 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:18:53.117588 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:53.120797 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:53.125061 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:53.155993 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:53.230648 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:53.246730 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:53.301635 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:53.345654 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:53.354790 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:53.377063 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:53.377827 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:53.395386 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:53.396755 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:18:53.397608 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:53.456290 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:18:53.560642 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:53.569773 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:53.582509 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:53.584430 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:53.594377 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:53.597822 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:53.610816 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:53.614681 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:53.636027 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:53.662992 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:53.681357 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:53.728643 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:53.738265 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:53.800673 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:53.808315 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:53.815505 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:53.816083 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:53.826790 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:53.840380 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:53.920454 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:53.986225 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:54.020231 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:54.037545 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:54.044494 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:54.047235 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:54.052304 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:54.077014 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:54.077030 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:54.102087 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:54.194651 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:54.196009 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:54.199425 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:54.211211 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:54.226806 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:54.245846 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:54.252131 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:54.271382 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:54.294448 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:54.326937 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:54.357066 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:54.402113 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:54.473846 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:18:54.493489 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:54.496058 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:18:54.501645 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:54.510954 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:54.511435 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:54.512001 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:54.516867 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:54.581302 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:54.583400 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:54.608997 32699 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:54.609205 32699 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:18:54.610121 32699 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:18:54.611689 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:54.612361 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:54.612475 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:54.612735 32699 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:18:54.612743 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:54.738960 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:54.739885 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:18:54.745105 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:54.747439 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:54.752017 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:54.762635 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:54.769551 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:54.795773 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:54.826424 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:54.886582 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:54.910989 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:54.956048 607 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:18:54.956240 607 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:18:54.956871 607 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:18:54.957445 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:18:54.958850 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:54.959229 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:54.959344 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:18:54.959892 607 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:18:54.985816 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:54.987998 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:54.989794 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:54.989861 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:54.990168 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:55.011060 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:55.033071 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 14:18:55.035032 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:55.078459 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:55.112557 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:55.114651 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:55.119447 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:18:55.190802 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:55.191462 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:55.206414 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:55.208667 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:55.216873 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:55.253151 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:55.276026 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:55.282986 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:55.370481 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:55.422502 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:18:55.426378 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:18:55.458628 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:55.463677 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:55.474310 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:55.478376 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:55.508227 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:55.560706 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 14:18:55.561939 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:18:55.571305 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:55.637248 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:55.689930 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:55.690451 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:55.711170 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:55.716732 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:18:55.726297 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:55.726701 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:55.740708 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:55.809986 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:55.840646 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:55.860184 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:18:55.938831 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:55.957798 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:55.958700 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:55.976425 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:55.976830 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:55.987682 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:56.003429 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:56.021538 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:56.040302 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:56.081961 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:18:56.108830 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:56.149986 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:56.155288 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:56.182940 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:56.194010 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:56.197010 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:56.201297 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:56.248831 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:56.249785 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:56.388217 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:56.390723 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:56.463845 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:56.542857 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:56.582084 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:56.606047 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:56.611433 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:18:56.634011 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:56.634444 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:56.642174 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:56.642778 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:56.648680 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:56.681805 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:56.701656 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:56.713930 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:56.766726 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:56.771476 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:56.794363 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:56.826231 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:56.837270 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240503 14:18:56.838558 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:56.841655 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:18:56.843114 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:56.851864 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:56.853513 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:56.866106 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:56.875501 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:56.911907 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:56.944489 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:56.974478 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:56.991690 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:57.022820 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:57.085669 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:57.103446 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:57.105469 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:18:57.111826 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:57.127929 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:57.135156 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:57.148761 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:57.152961 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:57.167598 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:57.191924 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:57.194176 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:57.243007 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:57.249419 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:57.324164 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:57.358830 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:57.363894 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:57.376542 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:57.395449 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:57.410574 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:57.518539 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:57.547739 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:18:57.618928 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:57.619381 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:57.620200 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:57.628454 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:18:57.630721 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:57.648838 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:57.730602 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:57.731660 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:18:57.738755 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:57.742849 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:57.765537 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:57.768110 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:57.830138 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:57.841037 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:57.851042 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:57.867970 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:57.871579 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:57.912593 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:58.050101 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:58.054481 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:58.068455 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:58.073760 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:58.077025 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:58.088337 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:58.125651 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:58.152745 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:58.174618 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:58.189224 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:58.288825 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:58.292133 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:58.306506 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:58.306973 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:58.307314 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:58.314303 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:18:58.347740 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:58.350560 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:18:58.456884 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:18:58.485175 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:58.532305 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:18:58.561841 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:58.580215 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:58.581542 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:58.599675 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:58.631580 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:58.666640 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:58.740612 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:58.755960 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:58.757179 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:58.762262 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:58.766552 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:58.785190 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:58.874758 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:58.939836 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:18:58.985714 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:59.004355 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:18:59.007470 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:59.008378 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:59.011806 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:59.012936 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:59.045801 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:59.100598 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:59.101969 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:59.106812 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:59.109077 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:18:59.184677 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:59.186540 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:59.207073 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:18:59.208279 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:59.224757 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:59.228482 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:59.230437 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:18:59.240347 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:18:59.259735 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:18:59.289829 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:59.300515 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:18:59.303604 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:59.355268 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:18:59.371776 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:18:59.445456 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:59.461516 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:18:59.478861 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:59.480875 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:59.482317 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:59.502413 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:18:59.506402 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:18:59.524698 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:18:59.536998 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:59.571055 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:18:59.594627 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:18:59.672387 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:18:59.674875 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:18:59.699280 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:59.712981 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:59.720172 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:18:59.724198 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:59.733978 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:18:59.743583 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:18:59.749460 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:18:59.751430 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:18:59.818395 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:59.897116 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:18:59.901430 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:18:59.903141 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:18:59.907275 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:18:59.922461 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:18:59.932464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:18:59.935251 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:00.010282 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:00.059242 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:19:00.076781 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:19:00.099143 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:00.139097 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:00.146087 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:00.152633 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:00.157127 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:00.168943 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:00.172953 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:00.187458 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:00.199034 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:00.214371 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:19:00.264086 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:19:00.356173 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:00.371280 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:00.397418 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:00.403057 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:00.404474 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:00.425470 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:00.434962 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 14:19:00.446131 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:00.449880 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:00.612958 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:00.627532 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:00.646958 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:19:00.656776 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:00.678467 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:00.795351 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:00.796450 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:00.802685 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:00.813529 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:00.818212 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:00.829892 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:00.957212 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:00.960093 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.000916 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:01.018641 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:01.038575 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:01.039574 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:01.043205 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:19:01.045130 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:01.060160 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:01.061200 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:19:01.091684 969 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:01.091883 969 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:01.092876 969 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:19:01.094504 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:19:01.094731 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:19:01.098166 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:01.098489 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:01.098584 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:01.099112 969 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:01.105758 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:01.115855 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:19:01.118695 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:01.127856 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.238579 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.239034 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:01.247495 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:19:01.250952 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:01.271286 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:01.277029 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:01.277626 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:01.338445 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:01.413424 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.463429 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:01.501178 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:19:01.507849 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:01.514365 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:01.521028 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:01.539786 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:01.541996 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:01.550000 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:01.574167 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:01.582819 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:01.638347 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.720052 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.724399 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:01.734175 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:01.746300 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:19:01.747721 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:01.749334 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:01.758374 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:01.758989 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:01.761803 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:01.774611 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:19:01.787330 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:01.790522 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:01.818584 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:01.860460 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:01.947144 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:01.960495 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:01.963791 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:01.969511 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:01.974897 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:02.007148 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:02.024200 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:02.041591 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:02.053117 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:19:02.053992 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:02.070690 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:02.078617 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:02.081650 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:02.147886 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:02.148355 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:19:02.172163 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:02.180611 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:02.191473 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:02.191493 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:02.196414 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:02.217100 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:02.262045 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:02.282195 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:02.309602 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:02.347353 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:19:02.407088 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:02.417241 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:02.429868 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:02.430879 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:02.435611 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:02.564343 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:02.581976 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:02.642908 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:02.735605 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:02.741499 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:02.763717 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:02.837601 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:19:02.847093 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:02.848420 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:02.876827 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:02.879554 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:02.883423 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:02.886317 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:02.898847 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:03.044149 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:03.056166 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:03.071004 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:19:03.075352 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:03.084086 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:03.092137 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:03.098358 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:03.113623 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:03.132023 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:03.178779 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:03.184748 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:03.301167 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:03.330624 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:03.345026 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:03.346501 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:03.352267 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:03.352749 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:03.377432 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:03.412470 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:19:03.414940 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:03.576082 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:03.576746 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:03.579452 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:03.583225 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:03.583868 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:03.604450 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:03.621168 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:03.659749 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:03.662171 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:03.682201 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:03.718333 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:03.803884 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:03.820096 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:03.823568 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:19:03.841269 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:03.844168 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:03.859603 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:03.861253 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:03.885743 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:03.908402 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:03.999737 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:04.020309 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:19:04.037456 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:04.041157 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:04.054054 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:04.056603 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:04.061050 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:04.061038 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:04.070295 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:19:04.088165 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:04.125617 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:04.129691 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:04.143821 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:04.203027 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:04.209779 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:04.245860 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:04.269857 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:04.272840 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:04.276535 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:04.285274 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:04.288913 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:04.292215 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:04.341073 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:04.387692 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:04.395503 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:19:04.454401 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:19:04.456430 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:04.485289 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:04.533166 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:04.552155 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:04.553980 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:04.555275 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:04.562383 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:04.574618 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:04.609130 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:04.614059 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:04.627122 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:04.678817 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:04.755173 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:04.774052 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:04.786033 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:04.821959 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:04.855795 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:04.953296 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:04.962247 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:04.978410 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:04.986311 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:04.990094 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:05.028266 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:05.062598 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:05.078626 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:05.102253 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:19:05.149693 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:19:05.183378 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:05.195372 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:05.197192 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:05.219491 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:05.220527 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:05.269125 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:19:05.275252 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:05.290872 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:05.302214 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:05.386533 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:05.396276 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:05.434799 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:05.434969 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:05.447425 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:05.461723 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:05.477052 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:05.526859 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:05.549055 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:05.586647 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:05.607921 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:05.616200 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:19:05.650039 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:05.663458 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:05.690564 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:05.696291 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:05.713086 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:05.730948 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:05.757930 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:05.775226 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:05.844010 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:19:05.888267 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:05.925282 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 14:19:05.959906 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:05.972698 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:05.981362 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:06.005079 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:06.008854 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:06.010147 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:06.020833 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:06.060853 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:06.083851 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:06.093096 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:19:06.204521 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:06.221972 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:06.229534 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:06.233054 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:06.238487 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:06.244360 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:06.244791 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:19:06.318586 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:06.324205 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:06.338892 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:06.388209 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:06.403393 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:06.422626 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:06.445791 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:06.446813 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:06.468147 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:06.477705 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:06.478741 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:06.493474 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:06.520594 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:06.572672 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:06.597177 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:06.662892 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:06.690383 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:06.691999 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:06.703617 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:06.703922 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:06.733623 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:06.734462 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:06.756215 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:06.797122 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:06.828387 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:06.903573 2997 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:06.903791 2997 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:06.904405 2997 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:06.906076 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:06.906244 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:06.906384 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:06.906661 2997 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:06.918305 969 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:06.918483 969 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:06.919310 969 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:19:06.920871 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:06.921020 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:06.921137 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:06.921388 969 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:06.921923 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:06.951341 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:06.987651 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:06.994427 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:07.003908 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:07.019173 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:07.038753 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:07.091993 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:19:07.118392 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:07.124642 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:07.127460 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:07.131729 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:07.145821 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:07.150986 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:07.172298 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:07.184993 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:07.201929 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:07.236277 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 14:19:07.237228 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:07.262643 2997 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:07.262816 2997 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:07.263435 2997 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:07.264871 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:07.265108 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:07.265221 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:07.265440 2997 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:07.314899 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:07.336548 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:07.339555 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:07.358721 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:07.360247 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:07.361411 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:19:07.362041 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:07.376751 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:07.400025 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:07.401978 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:07.501940 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:07.524448 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:07.565806 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:07.568133 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:07.571468 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:07.575163 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:07.577118 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:07.593147 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:07.723096 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:07.743602 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:07.744259 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:07.760082 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:07.804898 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:07.806074 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:07.808339 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:07.810094 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:07.814941 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:07.887480 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:07.893427 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:08.046730 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:08.063624 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:08.067909 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:08.081707 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:08.101208 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:08.108944 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:08.123090 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:08.143903 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:08.151542 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:08.158390 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:08.196709 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:08.235496 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:08.266753 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:08.275692 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:08.291891 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:19:08.298164 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:08.298797 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:08.307962 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:08.349572 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:08.351164 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:08.450034 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:08.506421 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:08.511426 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:08.519883 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:08.533859 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:19:08.556509 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:08.571722 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:08.584645 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:08.590909 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:08.745334 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:08.746690 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:08.756453 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:08.764138 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:08.764338 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:08.775779 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:08.780634 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:08.825753 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:08.835451 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:08.854985 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:08.870492 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:08.873926 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:08.907991 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:08.909317 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:08.914177 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:08.927875 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:09.030318 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:09.080103 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:09.103897 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:09.167284 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:09.169845 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:09.177206 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:09.177322 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:09.178464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:09.197851 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:19:09.199771 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:09.303331 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:09.330286 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:09.422673 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:09.422986 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:09.424459 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:09.426198 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:09.430227 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:09.434082 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:09.436196 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:09.449694 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:09.497757 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:09.523658 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:09.529541 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:09.557691 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:09.624616 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:09.633081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:09.635897 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:09.641352 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:09.642462 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:09.669549 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:09.684223 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:09.762291 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:09.767208 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:09.796196 3417 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:09.796432 3417 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:09.797389 3417 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:09.799293 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:09.799438 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:09.799568 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:09.799794 3417 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:09.820449 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:09.865291 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:09.867098 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:09.871918 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:09.892045 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:09.907073 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:09.932857 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:09.935838 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:09.947490 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:09.969986 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:09.973462 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:10.009152 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:10.012701 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:10.087708 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:10.090592 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:10.093353 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:10.101773 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:10.102725 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:10.136656 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:10.146143 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:19:10.204259 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:10.228621 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:10.231395 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:10.294544 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:10.307164 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:10.307909 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:10.314685 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:10.316540 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:10.319720 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:10.353834 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240503 14:19:10.359918 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:10.393436 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:10.476506 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:10.512776 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:10.553711 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:10.567919 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:10.570557 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:10.573863 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:10.578624 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:10.626178 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:19:10.627614 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:10.644347 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:10.677266 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:10.704658 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:19:10.720883 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:10.732424 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:10.747473 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:10.772477 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:10.777293 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:10.787833 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:10.797861 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:10.799587 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:10.819425 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:10.891023 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:11.027536 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:19:11.120229 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:11.131785 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:11.161702 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:11.166271 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:11.177742 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:11.182107 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:11.201654 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:11.202239 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:11.210530 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:11.236440 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:11.279225 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:11.317835 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:11.352479 3610 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:11.352690 3610 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:11.353472 3610 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:11.355281 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:11.366458 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:11.384027 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:11.384464 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:11.384757 3610 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:11.426139 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:11.440495 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:11.443675 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:11.449915 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:11.460340 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:19:11.461046 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:11.465552 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:11.505889 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:11.513278 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:11.544355 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:11.561517 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:11.670207 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:11.688038 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:11.691913 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:11.693289 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:11.701900 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:11.707517 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:11.718995 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:11.721421 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:11.725826 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:11.760779 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:11.762203 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:11.794145 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:11.795706 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:11.797976 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:11.868700 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:11.887737 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:11.906738 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:11.932024 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:11.932668 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:11.949591 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:11.968358 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:11.968786 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:11.982290 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:11.982671 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:12.113118 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:12.149791 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:12.166810 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:12.168505 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:12.190078 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:12.197866 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:12.208981 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:12.217262 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:12.228926 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:12.265488 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:19:12.266340 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:12.313391 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:12.319310 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:12.381582 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:12.403499 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:12.415293 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:12.422385 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:12.438567 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:12.438807 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:12.510107 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:12.590921 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:12.636281 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:12.661075 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:12.664119 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:12.678916 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:12.682400 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:12.682729 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:12.695585 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:12.707643 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:12.754949 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:12.873893 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:12.892159 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:12.892204 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:12.898068 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:12.908918 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:12.920650 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:12.925164 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:12.926038 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:13.032570 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:13.040572 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:13.076777 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:13.163749 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:13.183583 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:13.237850 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:19:13.270677 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:13.291426 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:13.318300 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:13.333652 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:13.335143 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:13.338150 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:13.344556 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:13.352737 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:13.398581 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:13.399682 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:13.457072 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:13.485884 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:13.584383 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:13.597084 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:13.597352 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:13.601202 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:13.603535 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:13.609988 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:13.628767 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:13.629478 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:13.699116 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:13.726209 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:13.755043 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:13.790521 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:13.811728 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:13.827562 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:13.850558 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:13.859961 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:13.874497 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:13.894644 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:13.894773 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:13.981534 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:13.994868 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:14.046972 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:14.053898 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:14.072257 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:14.074801 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:14.076431 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:14.093569 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:14.095927 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:14.096613 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:14.170193 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:14.174624 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:14.228438 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:14.239293 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:14.268093 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:14.289806 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:14.291163 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:14.303354 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:14.304322 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:14.308293 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:14.317082 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:14.336037 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:14.385183 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:14.385994 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:14.459779 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:14.520699 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:14.530558 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:14.530939 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:14.547472 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:14.547772 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:19:14.566421 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:14.577268 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:14.585875 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:14.634137 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:14.651826 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:14.707357 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:14.749531 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:14.772425 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:14.792193 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:19:14.792636 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:14.795956 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:14.796383 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:14.811635 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:14.825107 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:14.880723 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:14.924216 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:14.935786 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:14.951215 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:19:15.024864 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:15.025933 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:15.037300 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:15.037663 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:15.037837 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:15.059427 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:15.116804 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:15.129591 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:15.154376 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:15.190949 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:19:15.238201 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:15.379500 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:19:15.405894 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:15.443013 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:15.464336 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:15.494539 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:15.500939 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:15.505419 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:15.527268 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:15.528739 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:15.538749 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:15.544773 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:15.552767 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:15.553761 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:15.578491 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:19:15.606105 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:15.637537 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:19:15.703811 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:15.719417 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:15.726390 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:15.726459 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:15.748171 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:15.764405 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:15.835541 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:15.836493 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:15.915712 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:15.953891 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:15.963624 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:15.967453 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:15.968425 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:15.973906 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:15.979642 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:15.988987 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:16.012764 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:16.020061 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:16.020557 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:16.107210 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:16.168952 3646 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:16.169165 3646 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:16.170078 3646 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:19:16.172231 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:16.176254 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:16.176380 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:16.176716 3646 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:16.179100 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:19:16.191777 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:16.192961 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:16.211120 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:19:16.214354 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:16.217970 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:16.220321 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:16.225539 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:16.259444 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:16.408727 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:16.422339 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:16.428674 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:16.433617 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:16.448383 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:16.448736 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:19:16.453015 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:16.453194 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:16.454862 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:16.465056 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:16.532230 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:16.557637 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:16.586936 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:16.616057 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:16.651422 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:16.663837 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:16.666306 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:16.671355 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:16.672988 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:16.673883 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:16.699756 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:16.735997 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:16.736971 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:16.765803 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:16.768043 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:16.806982 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:16.827456 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:16.832279 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:16.925690 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:16.928694 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:16.941167 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:16.943429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:16.951375 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:16.965893 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:16.969107 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:16.977340 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:17.013475 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:17.022403 2367 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 56575.5us
W20240503 14:19:17.040653 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:17.109869 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:17.131141 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:17.171343 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:17.184335 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:17.190742 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:17.195719 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:17.205458 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:17.228592 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:17.235388 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:17.259912 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:17.285140 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:17.327224 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:17.356004 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:17.368661 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:19:17.544030 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:17.610494 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:17.616668 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:17.616868 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:17.617061 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:17.637629 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:17.645401 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:17.673188 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:17.739670 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:17.743165 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:17.747653 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:17.791040 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:19:17.808439 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:17.818220 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:17.829412 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:17.850001 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:17.860423 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:17.862069 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:17.901870 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:17.901960 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:17.944357 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:17.964901 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:17.983222 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:18.023290 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:18.071259 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:18.071707 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:18.090993 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:18.092933 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:18.115000 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:18.232575 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:18.259508 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:18.298379 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:18.317839 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:18.320016 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:18.323454 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:18.330070 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:18.336150 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:18.360889 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:18.372398 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:18.376348 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:18.431080 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:18.461467 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:18.506052 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:18.532974 3646 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:18.533192 3646 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:18.534263 3646 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:19:18.534869 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:19:18.535104 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:19:18.536597 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:18.536798 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:18.536926 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:18.537405 3646 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:18.583616 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:18.590615 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:18.591610 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:18.611725 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:18.616420 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:18.633908 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:18.685832 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:18.717911 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:18.785699 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:18.788102 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:18.788601 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:18.795841 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:18.799649 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:18.829608 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:18.840608 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:18.858592 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:18.860507 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:18.874354 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:18.901161 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:18.957325 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:18.989485 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:18.992578 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:19.014113 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:19.051250 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:19.062283 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:19.075500 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:19.084323 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:19.101763 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:19.105131 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:19:19.151618 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:19.175618 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:19.187260 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:19.257627 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:19.267979 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:19.291328 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:19.309181 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:19.328382 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:19.338725 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:19.345885 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:19.351915 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:19.371515 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:19.381681 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:19.392655 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:19.409111 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:19.550976 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:19.556145 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:19.558557 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:19.562882 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:19.628703 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:19.675402 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:19.692015 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:19.703505 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:19.763458 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:19.766543 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:19.767664 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:19.780149 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:19.798748 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:19.803158 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:19.803297 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:19:19.809566 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:19.813700 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:19.845638 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:19.977514 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:19.979885 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:19.994081 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:19.996198 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:20.006862 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:20.085314 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:20.089664 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:20.111797 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:20.127462 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:19:20.211416 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:20.213605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:20.216241 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:20.226097 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:20.226346 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:20.256129 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:20.296861 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 14:19:20.298274 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:20.315999 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:20.356740 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:20.378417 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:20.402033 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:20.430840 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:20.444854 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:20.448575 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:20.457386 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:20.459173 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:20.484683 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 14:19:20.532543 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:20.638844 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:20.667012 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:20.676611 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:20.693192 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:20.706689 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 14:19:20.707942 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:20.710866 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:20.714524 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:20.715454 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:20.721297 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:20.730712 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:20.777333 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:20.780025 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:20.794579 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:20.832042 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:20.931280 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:20.941303 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:20.941615 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:20.951956 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:20.953032 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:20.957403 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:20.959650 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:21.000164 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:21.048087 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:21.159066 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:21.177346 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:21.179708 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:21.183010 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:19:21.185124 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:21.192873 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:21.235107 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:21.262626 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:21.267879 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:21.306589 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:21.308706 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:21.419427 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:21.428397 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:21.435715 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:21.449544 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:21.457782 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:21.485284 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:19:21.497946 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:21.502413 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:21.512939 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:21.517105 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:21.519788 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:21.643352 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:21.663626 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:21.673440 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:21.705567 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:21.714697 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:21.735406 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:21.749481 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:21.912343 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:21.921957 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:21.923354 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:21.935719 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:21.937912 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:21.952544 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:21.975009 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:22.023010 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:22.061713 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:22.077677 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:22.080425 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:22.103932 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:22.118702 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:22.125532 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:22.127329 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:22.128063 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:22.129150 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:22.134383 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:22.151072 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:22.160459 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:22.181648 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:22.185268 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:19:22.344750 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:22.357868 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:22.364955 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:22.379370 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:22.408684 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:22.414631 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:22.435932 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:19:22.444392 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:19:22.445442 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:22.460251 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:22.466529 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:22.491257 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:22.493553 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:22.513733 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:22.513998 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:22.581270 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:22.592942 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:22.608701 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:22.608870 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:22.609545 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:22.623679 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:22.649513 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:22.776728 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:22.780843 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:22.823310 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:22.831349 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:22.850487 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:22.857832 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:22.886601 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:22.896667 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:23.019620 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:23.026189 3646 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:23.026461 3646 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:23.027675 3646 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:19:23.039095 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:23.039273 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:23.039402 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:23.039960 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:23.040222 3646 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:23.040481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:23.046567 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:19:23.055596 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:23.076395 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:23.076674 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:23.083163 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:23.110520 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:23.210789 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:23.236519 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:23.267011 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:23.294323 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:23.303452 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:23.306489 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:23.311678 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:23.312731 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:23.315528 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:23.328073 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:23.377588 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:23.381707 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:23.496803 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:23.502347 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:23.535984 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:23.565176 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:23.571447 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:23.589560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:23.600142 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:23.637005 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:23.660573 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:23.696276 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:23.699226 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:23.745270 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:23.754834 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:23.801337 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:23.943246 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:19:23.977612 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:23.979163 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:23.988338 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:23.990430 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:23.999941 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:24.013806 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:24.014782 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:24.032773 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:24.063482 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:24.092048 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:24.134382 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:24.139437 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:24.186247 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:24.196453 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:24.199425 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:24.206794 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:24.208961 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:24.219273 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:24.266417 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:24.266538 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:24.332463 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:24.379113 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:24.411935 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:24.421795 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:24.423537 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:24.435395 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:24.435480 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:24.440464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:24.448781 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:24.461721 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:24.512374 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:24.550086 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:24.584467 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:24.592105 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:24.593282 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:24.662129 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:24.669759 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:24.687466 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:24.689101 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:24.696403 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:24.712648 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:24.812970 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:24.834131 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:24.846709 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:24.852022 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:24.890349 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:24.937220 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:24.957216 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:24.958638 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:24.964097 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:24.978487 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:25.013792 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:25.031968 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:25.037055 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:25.082981 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:25.149900 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:25.162678 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:25.172374 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:25.175592 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:25.190343 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:25.204551 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:25.232964 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:19:25.235088 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:25.246182 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:25.301578 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 14:19:25.315243 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:25.381179 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:25.404220 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:25.421902 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:25.428962 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:25.429914 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:25.441653 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:25.446580 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:25.452754 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:25.484215 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:25.551645 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:19:25.567446 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:25.634325 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:25.682493 5560 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:25.682765 5560 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:25.683568 5560 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:19:25.688745 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:25.688890 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:25.705194 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:25.705379 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:25.706800 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:25.724092 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:25.726033 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:25.726148 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:25.735059 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:25.735261 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:25.735607 5560 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:25.736908 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:25.741552 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:25.769452 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:19:25.770467 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:25.802884 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:25.860807 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:25.971881 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:25.975157 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:26.001652 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:26.094802 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:26.116572 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:26.116571 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:26.135854 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:26.136955 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:26.143565 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:26.212170 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:19:26.245607 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:26.287361 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:26.326350 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:26.350905 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:26.354408 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:26.376670 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:26.402205 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:26.408421 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:26.412725 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:26.423774 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:26.432924 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:19:26.433336 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:26.441265 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:26.460414 6058 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:26.460637 6058 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:26.461385 6058 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:26.463033 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:26.474998 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:26.475170 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:26.475922 6058 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:26.510154 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:26.541834 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:26.640969 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:26.642028 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:26.649958 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:26.651304 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:26.665026 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:26.700188 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:26.705565 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:26.707530 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:26.725260 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:26.739311 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:26.761106 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:26.851130 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:26.851490 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:26.854804 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:26.877067 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:26.883380 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:26.886448 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:26.893216 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:26.919862 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:26.949069 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:26.955366 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:26.962186 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:26.980724 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:27.021070 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:27.084712 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:27.118799 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:27.145514 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:27.150616 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:27.153170 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:27.175158 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:27.187600 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:27.232008 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:27.254006 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:27.280087 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:27.340183 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:27.356048 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:27.366526 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:27.387005 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:27.388489 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:27.397804 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:27.399083 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:27.405004 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:27.469029 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 14:19:27.470760 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:27.572585 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:19:27.582875 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:27.598435 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:27.601337 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:27.614516 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:27.615970 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:27.619911 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:27.621685 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:27.664422 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:27.739264 3646 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:27.739452 3646 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:27.740320 3646 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:19:27.742105 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:27.742254 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:27.742363 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
W20240503 14:19:27.742491 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:27.742609 3646 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:27.766486 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:27.769400 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:27.776700 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:27.821408 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:27.827756 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:27.846107 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:27.848521 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:27.848568 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:27.897862 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:27.933012 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:27.933190 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:27.940605 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:27.979923 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:27.984159 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:27.986625 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:28.072196 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:28.139055 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:28.242337 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:28.252862 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:28.300493 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:28.305590 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:28.313321 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:28.328630 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:28.329634 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:28.354638 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:28.360297 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:28.437855 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:28.514180 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:28.561337 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:28.570972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:28.576769 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:28.579509 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:28.599980 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:28.600759 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:28.644356 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:28.665012 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:28.688267 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:28.691869 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:28.703687 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:28.749943 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:28.783218 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:28.807814 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:28.824517 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:28.825467 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:28.845464 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:28.865540 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:28.945794 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:29.009546 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:29.017611 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:29.037607 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:29.038195 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:29.046203 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:29.046607 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:29.096117 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:29.129931 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:29.132616 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:29.155432 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:29.195297 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:29.214936 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:19:29.244290 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:29.259006 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:29.272040 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:29.274948 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:29.276624 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:29.298434 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:29.298925 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:29.314350 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:29.316823 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:29.318315 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:29.342358 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:29.355078 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:29.423744 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:29.517778 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:29.538479 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:29.540148 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:29.543838 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:29.552964 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:29.559932 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:29.569619 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:29.570928 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:29.584944 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:29.658118 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:29.677925 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:29.698238 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:29.779781 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:29.795884 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:29.801213 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:29.806927 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:29.808235 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:29.843027 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:29.853266 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:29.886412 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:29.937986 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:30.015043 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:30.024969 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:30.026754 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:30.027357 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:30.045332 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:30.088795 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:30.089718 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:30.121791 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:30.139907 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:30.147110 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:19:30.187922 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:30.238121 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:30.304400 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 131 similar messages]
I20240503 14:19:30.321429 6433 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:30.321651 6433 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:30.322497 6433 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:19:30.323560 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:19:30.323802 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:19:30.324538 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:30.324668 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:30.324787 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:30.325037 6433 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:30.351655 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:30.363879 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:30.380065 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:30.450567 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:30.462543 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:30.494946 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:30.520771 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:30.536677 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:19:30.542824 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:30.554613 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:30.563441 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:19:30.572553 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:30.575366 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:30.597936 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:30.602906 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:30.621397 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:30.638919 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:30.642705 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:30.790266 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:30.808106 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:30.815210 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:19:30.830750 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:30.841641 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:30.846575 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:30.855222 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:30.866564 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:30.940656 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:30.959125 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:30.973778 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:31.015545 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:31.065641 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:31.086815 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:31.091624 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:31.092829 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:31.116797 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:31.248782 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:31.248934 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:31.271270 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:19:31.294209 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:31.296342 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:31.309064 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:31.328279 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:19:31.332314 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:31.335815 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:31.337661 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:31.352393 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:31.439103 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:31.451419 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:31.456207 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:31.567363 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:31.571962 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:31.573491 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:31.582562 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:31.583074 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:31.592764 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:31.597887 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:19:31.608142 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:31.628352 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:31.668004 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:31.675243 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:31.684825 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:31.791987 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:31.809391 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:31.811553 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:31.816208 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:31.822012 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:31.828045 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:31.848757 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:31.862608 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:31.912539 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:31.918722 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:31.969794 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:31.972126 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:31.975672 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:31.987249 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:32.051074 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:32.052222 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:32.057133 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:32.071977 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:32.074386 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:32.078621 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:32.079337 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:32.083493 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:32.123052 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:32.142031 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:32.263818 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:32.287014 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:32.294553 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:32.295228 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:32.308418 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:32.315646 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:32.334987 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:32.409333 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:32.433020 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:19:32.455058 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:32.479049 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:32.502624 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:32.599335 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:32.636878 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:32.689198 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:32.702315 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:32.710587 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240503 14:19:32.714322 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:32.729848 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:32.739558 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:32.743454 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:32.781505 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:32.787698 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:32.869292 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:32.875542 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:32.903434 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:32.910789 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:32.954986 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:32.958457 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:32.966984 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:32.968580 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:32.973881 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:32.975510 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:32.990084 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:32.991048 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:33.000173 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:33.105362 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:33.117447 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:33.196645 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:33.215353 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:33.216110 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:33.219692 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:33.229423 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:33.330503 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:33.360901 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:33.398985 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:33.408011 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:33.444108 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:33.449795 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:33.450472 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:33.455541 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:33.457710 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:33.476668 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:33.490614 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:33.572624 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:33.598232 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:33.668156 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:33.670902 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:33.689031 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:33.699256 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:33.705755 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:33.713073 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:33.718497 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:33.798815 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:33.808281 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:33.814862 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:33.908300 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:33.918231 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:33.931424 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:33.942572 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:33.944321 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:33.947634 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:33.962935 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:34.004529 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:34.020361 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:34.077111 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:34.137192 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:34.149727 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:34.155288 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:34.157033 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:34.158835 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:34.168673 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:34.168802 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:34.200129 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:34.249459 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:34.270316 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:34.277263 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:34.304198 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:34.331209 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:34.393250 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 14:19:34.396412 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:34.407578 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:34.423779 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:34.429560 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:34.431778 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:34.448588 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:34.458653 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:34.463907 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:34.471439 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:34.555714 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:34.622596 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:34.654970 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:34.676375 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:34.687269 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:34.704496 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:34.800143 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:34.816192 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:34.820791 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:19:34.858536 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:34.864231 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:34.870992 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:34.874768 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:34.909756 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:35.012059 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:19:35.034133 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:35.046686 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:35.115121 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:35.126044 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:35.133392 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:35.133704 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:35.138716 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:35.166693 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:35.183630 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:35.189626 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:35.221206 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:35.230660 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:35.258594 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:35.281447 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:35.314775 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 14:19:35.323660 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:35.343312 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:19:35.356791 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:35.360953 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:35.383131 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:35.395681 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:35.440119 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:35.514361 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:35.549413 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:35.549670 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:35.583392 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:19:35.618585 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:35.618973 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:35.659518 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:35.661522 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:35.677047 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:35.683543 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:35.689046 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:35.695063 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:35.719038 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:35.746641 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:35.809656 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:35.833063 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:19:35.834192 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:35.858935 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:35.935590 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:35.941239 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:35.941236 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:35.943385 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:35.946996 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:35.976693 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:36.028097 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:36.138361 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:36.153985 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:36.155692 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:36.202716 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:36.213899 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:36.214654 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:36.276643 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:36.288043 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:36.320932 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:36.356271 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:36.381295 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:36.387542 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:36.415524 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:36.428244 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:36.452312 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:19:36.457723 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:36.463398 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:36.469635 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:36.470932 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:36.497953 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:36.522393 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:36.590330 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:36.661406 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:36.667454 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:36.681635 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:36.684202 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:36.694345 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:36.695225 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:36.704075 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:36.722143 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:36.753386 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:36.771498 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:36.779300 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:36.843072 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:36.844444 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:36.912794 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:36.954175 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:37.029938 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:37.033991 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:37.037410 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:37.044837 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:37.089512 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:37.102622 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:37.116523 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:37.121555 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:37.146512 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:37.178263 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:37.194500 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:37.252689 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:37.271750 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:37.303048 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:37.315447 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:37.366050 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:37.372807 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:37.400611 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:37.425513 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:37.426070 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:37.431226 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:37.513386 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:37.518627 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:37.543237 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:37.571794 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:19:37.597355 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:37.633569 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:37.637629 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:37.641042 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:37.641196 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:37.659601 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:37.664970 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:37.683674 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:37.684810 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:37.696206 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:37.724853 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:37.729733 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:37.782255 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:37.795172 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 14:19:37.882457 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 14:19:37.941813 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:37.951154 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:37.965232 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:37.979440 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:37.985749 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:38.003938 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:38.088060 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:38.115012 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:38.146658 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:38.161989 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:38.163866 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:38.173852 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:38.196033 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:38.197242 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:38.207885 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:38.214984 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:38.215579 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:38.243286 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:38.243768 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:38.391713 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:38.421177 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:38.428675 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:38.441788 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:38.449278 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:38.456151 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:38.562505 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:38.576035 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:38.597270 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:38.642958 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:38.651306 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:38.666088 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:38.667055 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:38.695981 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:38.700325 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:38.763029 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:38.776788 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:38.797588 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:38.843827 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:19:38.866238 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:38.886467 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:38.896634 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:38.897480 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:38.898903 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:38.923080 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:38.964640 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:38.973766 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:39.014868 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:39.056772 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:39.073850 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:39.083544 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:39.098731 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:39.168987 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:39.186247 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:39.217730 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:39.307214 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:39.312767 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:39.318583 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:39.320297 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:39.327391 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:39.353549 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:39.361318 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:39.380666 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:39.429632 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:39.485462 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:39.570525 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:39.573856 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:39.576527 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:39.586158 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:39.586433 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:39.601112 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:39.601387 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:39.625015 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:39.676393 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:39.677671 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:39.712788 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:39.725718 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:39.823909 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:39.840821 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:39.841758 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:39.847656 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:39.863348 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:39.869176 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:39.886123 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:39.931505 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:39.935066 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:39.949054 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:19:39.959659 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:40.005049 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:40.061584 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:40.074719 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:40.078524 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:40.097023 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:40.097589 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:40.099937 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:40.100968 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:40.116750 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:40.164948 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:40.180845 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:40.194031 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:40.203405 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:40.267174 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:40.306596 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:40.315156 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:40.315110 7812 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:40.315608 7812 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:40.316578 7812 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:19:40.318908 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:40.323626 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:40.326452 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:19:40.327984 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:40.332679 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:40.332873 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:40.333318 7812 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:40.339445 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:40.342973 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:40.365976 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:40.458910 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:40.459616 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:40.475054 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:40.546670 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:40.547928 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 14:19:40.554901 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:40.560963 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:40.573738 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:40.582643 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:40.583297 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:40.594976 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 102 similar messages]
I20240503 14:19:40.595202 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:40.596171 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:40.777925 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:40.782402 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:40.815495 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:40.816105 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:40.828056 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:40.840070 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 14:19:40.845036 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:40.850167 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:40.860656 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:40.880110 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:41.027539 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:41.049436 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:41.060096 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:41.065752 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:41.076907 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:41.083899 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:41.105834 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:41.132771 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:41.143498 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:41.287415 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:41.311439 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:41.333709 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:41.337710 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:41.397090 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:41.402386 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:41.434191 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:41.456316 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:41.485015 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:41.493017 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:41.502362 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:19:41.502992 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:41.504153 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:41.521919 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:41.527945 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:41.544518 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:41.647827 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:41.677083 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:41.704257 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:41.713057 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:41.717598 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:41.717679 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:41.736331 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:41.749720 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:41.751901 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:41.773429 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:41.871095 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:41.925158 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:41.953991 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:41.955879 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:41.963358 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:41.964069 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:41.976858 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:41.983659 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:41.983669 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:42.029433 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:42.043059 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:42.085099 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:42.111598 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:42.116900 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:42.174798 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:42.186369 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:42.191685 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:42.194619 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:42.208971 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:42.213435 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:42.246007 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:42.350540 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:42.357427 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:42.384184 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:19:42.435745 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:42.461870 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:42.463104 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:42.478264 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:42.479781 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:42.491904 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:42.522011 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:42.525211 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 14:19:42.553472 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:42.554677 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:42.563160 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:42.565439 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:42.583369 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:42.597414 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:42.650538 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:42.655493 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:42.680666 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:42.703471 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:42.703706 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:42.720068 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:42.730093 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:42.733675 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:42.756659 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:42.766650 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:42.805853 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:42.838003 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 14:19:42.891139 7824 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:42.891350 7824 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:42.892304 7824 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:19:42.893286 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:19:42.893507 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:19:42.896445 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:42.910532 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:42.910718 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:42.911130 7824 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:42.931838 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:42.937186 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:42.956219 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:42.966295 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:43.005271 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:43.067287 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:43.090215 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:43.120863 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:43.123257 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:43.149495 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:43.161504 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:43.177686 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:43.188529 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:43.191983 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:43.208189 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:43.209278 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:43.218230 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:43.231602 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:43.238507 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:43.547638 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:43.617038 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:43.624954 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:43.643067 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:43.643054 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:43.651259 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:43.659479 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:43.663946 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:43.686283 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:43.697765 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:43.755112 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:43.819924 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:43.837824 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:43.843665 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:43.852304 8325 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:43.852495 8325 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:43.853166 8325 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:43.856793 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:43.859752 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:43.863335 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:43.876399 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:43.877795 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:43.878013 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:43.878331 8325 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:19:43.880089 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:43.887794 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:44.012216 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:44.054862 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:44.088472 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:44.090960 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:44.093726 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:44.094907 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:19:44.095225 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:44.114758 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:44.125262 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:44.141450 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:44.143429 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:44.159670 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:44.185046 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 14:19:44.190752 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:44.231766 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:44.278956 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:44.280874 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:44.301069 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:44.315860 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:44.316599 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:44.324359 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:44.334146 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:44.343683 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:44.456733 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:44.492344 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:44.517426 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:44.566617 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:44.573814 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:44.579631 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:44.595777 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 14:19:44.607846 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:44.614583 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:44.617085 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:44.666254 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:44.736656 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:44.787693 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:44.791141 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:44.802721 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:44.805168 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:44.813966 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:44.822717 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:44.823940 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:44.840435 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:44.856971 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:44.861964 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:44.903651 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:44.920722 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:44.922669 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:44.960151 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:44.999868 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 14:19:45.073951 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:45.086032 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:45.097347 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:45.104176 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:45.138815 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:45.140286 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:45.144905 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:45.189633 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:45.206636 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:45.210664 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:45.225498 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:45.323537 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:45.328294 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:45.332880 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:45.344683 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 130 similar messages]
I20240503 14:19:45.359117 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:45.389812 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:45.399935 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:45.401324 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:45.418310 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:45.426765 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:45.525430 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:45.589107 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:45.640666 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:19:45.641638 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:45.666024 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:45.708817 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:45.719815 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:45.760574 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:45.801306 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:45.808100 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:45.822167 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:45.822301 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:45.823137 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:45.862130 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:45.884405 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:45.956967 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:19:46.005456 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:19:46.042868 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:46.059907 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:46.060695 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:46.062979 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:46.090941 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:46.139775 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:46.259436 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:46.259871 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:46.275631 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:46.283531 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:46.285557 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:46.288165 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:46.295235 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:46.307904 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:46.312892 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:46.321324 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:46.370308 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:46.431762 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:46.513813 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:46.518502 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:46.550241 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:46.552201 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:46.560207 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:46.572847 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:46.579238 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:46.589745 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:46.598134 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:46.605335 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:46.616747 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:46.663594 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:46.727632 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:46.754050 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:46.768931 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:46.779393 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:46.791862 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:46.809028 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:46.809114 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:46.809737 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:46.827657 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:19:46.857544 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:46.867594 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:46.880776 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:46.889811 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:46.912820 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:46.934721 7824 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:46.934919 7824 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:46.935739 7824 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:19:46.937474 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:46.937669 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:46.937785 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
W20240503 14:19:46.937908 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:46.938002 7824 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:46.973821 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:47.025560 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:47.035605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:47.035763 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:47.056676 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:47.057569 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:47.064757 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:47.140934 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:47.219834 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:47.250725 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:47.278777 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:47.288713 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:47.289906 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:47.312891 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:47.321614 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:47.325870 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:47.334935 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 14:19:47.365092 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:47.402143 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:47.404654 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:47.464032 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:47.468905 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:47.481374 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:47.521636 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:47.526165 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:47.536326 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:47.566891 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:47.573411 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:47.597764 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:47.610318 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:47.629534 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:47.653288 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:47.700670 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:47.712687 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:47.787621 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:47.804908 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:47.824772 8325 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:47.824976 8325 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:47.825608 8325 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:47.827109 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:47.827350 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:47.827472 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:47.827700 8325 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:47.844703 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:47.886418 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:47.921900 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 14:19:47.934365 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:47.941601 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:47.999979 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:48.006171 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:48.020624 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:48.040562 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:48.043627 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:48.047256 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:48.067730 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:48.088287 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:48.110025 8325 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:48.110242 8325 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:48.110992 8325 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:48.113672 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:48.113859 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:48.113960 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:48.114233 8325 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:48.196056 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:48.242386 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:48.249863 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:48.267279 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:48.270328 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:48.276127 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:48.358911 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:48.363508 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:48.370528 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:48.386150 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:48.386551 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:48.480717 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:48.483605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:48.494232 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:48.497130 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:48.525394 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:48.526769 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:48.629182 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:48.720062 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:48.720522 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:48.739706 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:48.767593 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:48.777745 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:48.786234 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:48.791893 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:48.897432 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:48.905079 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:48.925504 7824 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:19:48.925704 7824 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:19:48.926591 7824 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:19:48.927438 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:19:48.927707 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:19:48.928853 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:48.929087 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:48.929219 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:19:48.929495 7824 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:48.947546 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:48.950273 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:48.956913 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:48.972204 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 14:19:48.988471 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:48.991596 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:49.006954 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:49.009363 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:49.014047 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:49.015367 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:49.018997 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:49.019433 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:49.093036 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:49.118741 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:49.127266 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:49.192412 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:49.233132 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:49.243432 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:49.256501 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:19:49.262593 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:49.269027 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:49.287207 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:49.291958 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:49.298972 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:49.300127 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:49.364133 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:49.370726 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:49.438202 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:49.500447 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:49.519373 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:49.528923 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:49.529055 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:49.530124 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:49.544778 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:49.570147 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:49.634538 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:49.647707 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:49.692099 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:49.708233 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:49.743147 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:49.758492 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:49.761214 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:49.765975 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:49.766491 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:49.793051 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:49.820653 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 14:19:49.829733 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:49.870044 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:49.893860 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:49.929749 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:49.939815 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:49.952775 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:50.003477 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:50.070138 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:50.126637 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:19:50.188531 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:50.208041 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:50.216862 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:50.219336 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:50.223205 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:50.230147 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:50.241565 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:50.297019 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:50.335398 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:50.353911 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:19:50.355227 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:50.393456 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:50.409858 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:50.418133 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:50.419955 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:50.422394 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:50.432969 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:50.456964 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:50.461957 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:50.466393 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:50.569365 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:50.625576 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:50.641360 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240503 14:19:50.645462 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:50.649014 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:50.663906 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:50.676478 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:50.680379 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:50.688509 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:50.697538 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:50.713339 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 14:19:50.715732 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:50.892805 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:50.895921 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:50.909337 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:50.913014 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:50.934450 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:50.937316 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:50.944892 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:50.957652 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:50.987665 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:51.064102 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240503 14:19:51.132516 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:51.141448 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:51.152382 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:51.158592 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:51.159019 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:51.213647 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:51.274022 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:51.354333 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:51.356912 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:51.361621 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:51.378849 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:51.397261 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:51.398203 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:51.415269 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:51.438297 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:51.459899 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:51.499097 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:51.500005 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:51.504621 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:51.517813 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:51.539280 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:51.576031 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:51.588824 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:51.597265 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:51.608461 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:51.610819 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:51.662072 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:51.663800 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:51.743459 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:51.747716 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:51.775847 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:51.815742 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:51.823038 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:51.837495 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:51.840307 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:51.841825 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:51.844327 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:51.873394 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:51.875766 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:51.905558 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:51.910871 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:51.946439 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:51.971717 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:19:51.994338 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:52.047895 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:52.095074 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:52.133296 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:52.193841 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:52.233191 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:19:52.277386 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:52.282294 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 14:19:52.289629 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:19:52.295248 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:52.301524 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:52.302621 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:52.307402 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:52.374212 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:52.387576 2628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 71595.3us
W20240503 14:19:52.391469 2626 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.2:43457, user_credentials={real_user=slave}} blocked reactor thread for 79203.2us
W20240503 14:19:52.402688 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:52.437034 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:52.454766 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:52.499413 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:52.507755 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:52.579346 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:52.590970 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:52.591714 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:52.593461 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:52.610775 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:52.614787 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:52.659004 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:52.662518 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:52.670842 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:52.733307 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:52.783248 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:52.836455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:52.837551 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:52.850359 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:52.852316 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:52.855612 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:52.884228 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:52.910424 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:52.922240 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:52.974789 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:52.989516 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:53.011988 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:53.046733 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:53.061885 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:53.063279 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:53.066231 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:53.072041 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:53.072656 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:53.160169 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:53.169260 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:53.211623 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:53.280403 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 55 similar messages]
I20240503 14:19:53.283214 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:53.284137 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:53.291819 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:53.311132 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:53.316370 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:53.316741 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:53.404513 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:53.452740 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:53.465272 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:53.503154 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:53.511957 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:53.531682 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:53.535506 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:53.543447 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:53.555307 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:53.687866 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:53.715554 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:53.737514 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:53.763068 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:53.766674 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:53.768704 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:53.769482 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:53.770855 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:53.773919 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:53.822304 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:53.825186 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:53.837512 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:53.890427 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:53.922977 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:53.969977 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:53.977744 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:53.983934 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:19:53.984463 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:54.002779 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:54.005551 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:54.012040 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:54.083447 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:54.164281 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 14:19:54.243863 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:54.278817 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:54.279450 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:54.289381 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:54.322546 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:54.337474 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:54.400631 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:54.404147 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:54.411473 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:54.420150 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:54.443565 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:54.449159 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:54.450227 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:54.479202 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:54.480793 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:54.536625 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:19:54.593524 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:54.608794 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:54.658288 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:54.660003 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:19:54.669909 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:54.670928 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:54.697832 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:54.700421 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:54.705523 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:54.713459 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 14:19:54.738207 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:54.749204 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:54.820798 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:54.846015 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:54.853410 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:54.892567 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:54.927917 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:54.935926 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:54.944245 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:54.971450 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:54.971685 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:54.991925 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:55.041661 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:55.137845 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:55.148569 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:55.160112 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:55.168344 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:55.171525 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:55.173945 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:55.177120 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:19:55.202533 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:55.220230 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:55.274881 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:55.340371 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:55.343416 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:55.374559 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 14:19:55.410923 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:55.412428 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:55.416808 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:55.422089 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:55.426909 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:55.427793 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:55.448324 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:55.506819 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:55.535048 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:55.536507 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:55.567947 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:55.612394 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:55.627841 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:55.629200 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:55.637749 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:55.670392 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240503 14:19:55.683460 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:55.698359 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:55.712682 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:55.729435 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:55.779117 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:55.819783 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:55.876775 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:55.899804 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:55.901491 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:55.902804 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:55.908584 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:55.925015 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:55.933266 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:55.950785 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:56.009307 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:56.073174 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 14:19:56.133756 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:56.146090 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:56.149055 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:56.157251 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:56.161962 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:56.270049 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:56.284513 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:56.322914 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:56.336977 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:56.360071 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:56.370383 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:56.395897 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:19:56.396214 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:56.408368 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:56.410799 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:56.444676 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:56.550077 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:19:56.562891 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:56.581833 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:56.585122 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:56.591581 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:56.617990 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:56.682268 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:56.707028 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:56.744939 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:56.748821 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:56.752453 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:56.753603 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:56.797205 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:56.809047 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:56.820188 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:56.823521 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:56.828127 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:56.846881 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:56.895594 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:56.901048 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:56.947240 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:57.020262 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:57.037626 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:57.040936 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:57.045107 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:57.047571 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:57.052131 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:57.052778 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:57.097595 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:57.125161 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:57.136773 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:19:57.137707 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:19:57.164402 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:57.189726 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:57.230309 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:19:57.261453 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:57.293116 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:57.297345 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:57.301220 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:57.320071 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:57.367877 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:57.388686 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:57.391781 2368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 64227us
W20240503 14:19:57.401666 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:57.434806 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:57.451936 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 14:19:57.506618 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:57.535591 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:57.563756 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:57.563948 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:57.572089 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:57.572551 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:57.578998 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:57.580428 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:57.595288 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:57.613328 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:57.734347 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:57.779547 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:57.786285 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:57.788339 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:57.827536 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:57.827900 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:57.834674 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:57.837172 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:57.917801 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:57.918869 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:57.943131 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:57.950091 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:57.985610 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:58.027619 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:58.061787 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:58.066843 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:19:58.074059 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:58.098659 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:58.099083 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:58.099980 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:58.149399 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:58.169596 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:58.175278 10552 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:58.175478 10552 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:19:58.176292 10552 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:58.177762 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:58.178180 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:58.178325 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:58.178629 10552 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:58.292361 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:58.292984 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:58.299973 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:58.306635 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 14:19:58.325188 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:58.330248 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:58.345894 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:58.359617 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:58.412806 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:58.437419 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:58.551420 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:58.650192 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:58.681087 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:58.727531 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:58.756999 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:58.757624 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:58.767983 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:58.772387 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:19:58.773527 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:58.774988 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:58.776284 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:58.800299 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:19:58.810010 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:19:58.810863 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:19:58.812565 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:58.907068 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:19:58.923416 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:19:58.981406 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:58.989859 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:59.000519 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:59.014547 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:59.020928 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:59.046917 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:59.056133 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:59.153684 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:59.174850 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:59.225235 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:59.225952 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:59.226440 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:59.228569 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:59.255790 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:59.258126 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 14:19:59.262934 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:19:59.281077 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:19:59.343752 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:19:59.366762 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:59.368290 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:59.383349 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:19:59.406626 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:59.450985 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:59.468932 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:59.472040 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:59.480305 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:59.483565 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:19:59.491655 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:19:59.503315 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:19:59.543846 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:19:59.569772 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:19:59.615917 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:19:59.677628 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:19:59.701542 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:19:59.701642 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:19:59.719805 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:19:59.735857 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:59.750111 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:59.755589 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:59.776849 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:19:59.868474 10864 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:19:59.868659 10864 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
W20240503 14:19:59.869089 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 14:19:59.869321 10864 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:19:59.870668 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:19:59.873528 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:59.873651 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:19:59.874034 10864 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:19:59.904475 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:19:59.929055 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:19:59.965698 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:19:59.968601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:19:59.981616 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:19:59.981967 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:19:59.987773 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:19:59.995699 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:19:59.998426 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:00.006595 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:20:00.024027 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:00.064738 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:00.094451 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:00.175979 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:00.203439 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:00.204190 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:00.204531 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:00.212261 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:00.235296 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:00.279934 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:00.284655 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:20:00.314599 10864 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:00.314775 10864 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:00.315502 10864 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:00.316839 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:00.316995 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:00.317102 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:00.317354 10864 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:00.321281 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:00.328720 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:00.357203 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:00.357779 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:00.386554 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:00.395167 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 128 similar messages]
I20240503 14:20:00.473145 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:00.477164 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:00.482689 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:00.489570 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:20:00.490155 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:00.503614 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:00.527925 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:00.578881 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:00.646442 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:00.666106 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:00.700032 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 14:20:00.900131 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 14:20:00.908901 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:00.916517 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:00.926705 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:00.927415 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:20:00.943244 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:00.961103 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:00.961162 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:00.967510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:00.976575 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:01.095397 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240503 14:20:01.112145 10274 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:01.112367 10274 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:01.113284 10274 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:20:01.116179 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:01.117903 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:01.119694 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:01.119863 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:01.120167 10274 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:01.122642 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:01.130658 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:01.161211 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:01.170105 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:01.181752 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:01.183262 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:01.190948 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:01.214150 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:01.215596 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:01.239502 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:01.284664 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:20:01.365933 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:01.375893 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:20:01.422798 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:01.434856 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:01.437714 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:01.438501 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:01.441952 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:01.448436 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:01.448571 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:01.499025 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:20:01.529382 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:01.545933 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:01.557646 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:01.651031 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:01.653618 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:01.655124 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:01.661374 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:01.682711 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:01.701632 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:01.711141 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:01.755641 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:20:01.901011 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:01.940132 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:01.940467 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:01.946069 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:01.960412 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:01.970571 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:01.978595 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:01.986543 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:02.001471 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:02.009737 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:20:02.061201 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:02.086272 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:02.095342 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:20:02.141433 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:02.152637 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:02.159757 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:02.162334 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:02.166558 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:02.175887 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:02.183522 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:20:02.184636 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:02.185079 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:02.212009 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:02.307734 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:02.316048 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:02.337013 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 14:20:02.401898 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:02.404708 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:02.411463 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:02.416872 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:02.418929 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:02.431100 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 14:20:02.432116 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:02.441798 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:02.443359 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:02.621299 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:02.630045 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:02.641866 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:02.644330 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:02.652693 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:02.655079 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:02.661064 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:02.689172 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:02.689929 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:02.714237 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:02.728329 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:02.733173 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:02.779058 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:02.798185 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:02.883275 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:02.959362 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:03.026120 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:03.066744 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:03.069978 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:03.077901 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:03.089300 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:03.091668 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:03.096700 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:03.166534 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:03.185087 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:03.196178 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:03.223907 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:03.296571 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:03.300633 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:03.304299 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:03.336321 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:03.336396 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:03.349406 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:03.376732 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:20:03.448709 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:03.492448 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:20:03.496592 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 14:20:03.584147 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:03.591352 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:03.602476 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:03.603029 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:03.613077 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:03.645138 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:03.664942 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:03.675688 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:03.700904 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:03.744959 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:03.748953 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:03.785796 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:03.823848 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:03.846953 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:03.858139 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:03.861941 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:03.877772 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:03.879297 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:03.880230 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:03.939222 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:03.944320 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:20:04.069208 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:04.080533 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:04.097836 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:04.102950 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:04.104336 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:04.116654 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:04.120218 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:04.138455 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:04.294377 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:04.294868 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:04.300092 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:04.308570 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:04.311426 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:04.328491 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:04.342116 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:04.348259 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:04.433770 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:04.445050 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:04.475399 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:04.534350 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:04.560051 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:04.562690 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:04.566972 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:04.568723 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:04.576148 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:04.595650 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:04.615621 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:04.633539 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:04.646267 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:20:04.648856 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 14:20:04.666170 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:04.725651 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:04.785981 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:04.807507 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:04.810545 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:04.810667 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:04.828379 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:04.855424 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:04.889242 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:04.893249 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:04.901772 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:04.917124 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:04.917745 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:04.931403 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:04.937355 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:05.005714 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:05.011492 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:05.015369 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:05.041486 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:05.102527 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:05.142361 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:05.215893 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:05.230886 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:05.256987 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:05.259043 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:05.278635 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:05.310788 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:05.344516 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:05.429442 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:05.432837 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240503 14:20:05.451773 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:05.462205 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:05.478056 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:05.481334 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:05.509828 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:05.518422 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:05.538036 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:05.538311 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:05.572540 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:05.638799 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:05.695441 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:05.750397 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20240503 14:20:05.782366 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:05.793980 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:05.795894 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:05.801944 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:05.806180 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:05.847710 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:05.856436 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:05.929745 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:05.936414 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:06.003566 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:06.003610 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:20:06.012132 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:06.017823 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:06.039319 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:06.039311 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:06.046007 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 14:20:06.073695 11891 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:06.073881 11891 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:06.074815 11891 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:20:06.076197 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:20:06.076438 2301 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:20:06.076642 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:06.083029 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:06.084102 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:06.084518 11891 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:06.084733 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:06.100539 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:20:06.148157 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:06.219050 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:06.249241 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:06.251533 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:06.254081 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:06.255404 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:06.258226 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:06.258496 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:06.266045 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:06.271631 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:06.302459 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:06.408443 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:06.417729 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:06.452898 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:06.469062 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:06.469331 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:06.483899 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:06.487546 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:06.491241 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:06.506805 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:06.508152 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:06.525269 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:06.590406 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:06.679781 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:06.697970 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:06.705150 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:06.715394 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:06.741714 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:06.770653 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:06.778002 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:06.806047 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:06.875762 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 14:20:06.913834 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:06.957480 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:06.962447 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:06.969643 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 14:20:06.984534 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:06.999356 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:07.003163 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:07.017526 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:07.037173 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:07.098335 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:07.104779 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:07.124758 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:07.161166 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:07.195597 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:07.242143 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:20:07.267920 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:07.273828 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:07.319715 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:07.326093 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:07.333704 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:07.396283 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:07.422454 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:07.445900 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:07.456257 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:07.457540 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:07.458307 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:07.463116 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:07.468079 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:07.540884 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:07.570443 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:07.625339 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:07.660851 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:07.668960 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:07.670192 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:07.678928 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:07.681663 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:07.683930 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:07.690407 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:07.696660 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:07.724799 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:07.741421 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:07.754218 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:07.783541 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:07.886437 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:07.893043 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:07.905076 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:07.925272 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:07.926095 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:20:07.937155 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:08.074342 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:08.075564 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:08.108021 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:08.111011 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:08.145460 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:08.157421 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:08.162731 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:08.164251 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:08.164909 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:08.168674 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:08.170164 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:08.170595 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:08.175513 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:08.234889 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:08.411955 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:08.412204 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:08.423686 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:08.431269 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:08.451947 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:08.453881 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:08.457441 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:08.518813 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:08.547732 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:08.549988 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:20:08.578328 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:08.596952 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:08.647251 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:08.648836 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:08.668658 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:08.672463 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:08.674808 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:08.689877 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:08.695042 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:08.779913 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:08.830066 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:08.837392 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:20:08.879783 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:08.898533 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:08.907788 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:08.911545 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:08.911593 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:08.912891 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:08.938324 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:08.992798 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:09.003223 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:09.007861 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:09.023373 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:09.074376 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:09.120239 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:09.129848 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:20:09.134953 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:09.141649 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:09.147447 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:09.165680 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:09.166702 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:09.289686 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:09.318511 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:09.346725 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 14:20:09.385530 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:09.395469 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:09.440096 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:09.475255 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:09.526242 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:09.549350 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:09.563338 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:09.567265 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:09.575341 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:09.582003 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:09.582662 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:09.586067 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:09.611678 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:09.617023 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:09.716899 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:09.768446 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:09.768822 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:09.783334 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:09.792475 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:09.793618 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:09.803030 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:09.812973 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:09.893759 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:09.917068 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:09.925779 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:09.934072 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:09.968381 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:09.987720 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:09.990128 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:10.014036 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:10.021111 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:10.033172 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:10.056147 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:10.067945 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:10.084645 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:10.123001 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:10.173808 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:20:10.265178 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:10.267473 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:10.314827 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:10.317773 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:10.318444 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:10.349489 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:10.350343 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:10.382437 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:10.449553 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:10.455389 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:10.479287 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:10.506649 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:10.511252 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:10.528606 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:10.538573 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 131 similar messages]
I20240503 14:20:10.539772 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:10.553843 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:10.562480 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:10.562862 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:10.611344 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:10.618908 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:10.628321 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:10.741703 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:20:10.768568 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:10.803122 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:10.822005 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:10.824154 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240503 14:20:10.824820 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:10.825551 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:10.829823 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:10.866693 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:10.999647 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:11.011957 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:11.029181 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:11.052129 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:11.053864 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:11.058689 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:11.068431 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:11.080114 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:11.112066 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:11.134102 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:11.134274 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:20:11.154531 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:11.160898 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:11.249966 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:11.254815 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:11.260627 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:11.282193 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:11.284494 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:11.308635 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:11.322304 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:11.338265 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:11.350700 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:11.545132 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:11.554262 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:11.587574 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:11.612985 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:11.626616 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:11.644408 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:11.673010 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:11.712839 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:11.724540 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:11.730347 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:11.736686 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:11.746955 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:11.759532 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:11.775883 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:11.791364 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 14:20:11.815376 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:11.931721 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:11.945130 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:11.950760 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:11.955263 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:11.973575 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 14:20:11.974272 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:11.982856 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:11.997942 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:12.015918 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:12.023802 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:12.040119 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:12.074278 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:12.094084 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:12.119992 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:12.124100 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:12.162272 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:12.197984 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:12.202533 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:12.214197 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:12.216048 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:12.232561 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:12.318156 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:12.345225 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:20:12.363637 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:12.435117 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:12.444026 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:12.446360 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:12.454797 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:12.478991 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:12.503710 11388 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:12.503898 11388 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:12.504848 11388 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:20:12.506844 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:20:12.507073 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:20:12.508601 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:12.521212 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:12.521492 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:12.521772 11388 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:12.531494 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:12.590240 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:12.591742 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:12.609315 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:12.609933 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:12.649672 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:20:12.676669 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:12.687664 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:12.689741 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:12.694633 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:12.715821 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:12.730346 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:12.750481 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:12.870077 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:12.876562 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:12.889298 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:12.911697 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:12.921293 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:12.932767 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:12.933012 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:12.938937 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:12.953246 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:12.955466 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:12.978118 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:12.993286 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:13.083774 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:13.118592 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:13.165246 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:13.178023 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:13.181938 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:13.186419 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:13.188244 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:13.189888 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:13.196388 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:13.203470 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:13.207325 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:13.308864 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:13.322283 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:13.397343 12861 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:13.397528 12861 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:13.398216 12861 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:13.401507 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:13.402348 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:13.414688 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:13.415433 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:13.431308 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:13.436419 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:13.447849 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:13.448019 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:13.448302 12861 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:13.474231 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:13.504185 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:13.593994 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:20:13.614516 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:13.615679 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:13.660112 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:13.683724 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:13.697903 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:13.719827 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:13.812229 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:13.813675 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:20:13.832912 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:13.832960 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:13.839138 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:13.853536 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:13.903563 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:13.905129 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:13.919384 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:14.004042 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:14.023978 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:14.031875 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:14.032490 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:14.035212 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:14.041088 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:14.066669 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:14.204533 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:14.209764 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:14.278126 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:14.283679 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:14.287422 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:14.290308 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:14.301323 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:14.315251 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:14.321290 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:14.333423 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 14:20:14.371220 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:14.384456 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:14.412072 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:14.485329 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:14.524123 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:14.528092 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 14:20:14.531585 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:14.535495 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:14.541285 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:14.558419 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:14.561520 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:14.562336 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:14.576457 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:14.635226 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:14.641168 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:14.644515 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:14.661701 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:14.715677 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:14.718590 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:14.723047 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:14.791791 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:14.795521 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:14.801090 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:14.817461 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:14.822876 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:14.837689 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:14.947373 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:15.002446 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:15.003430 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:15.012148 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:15.012970 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:15.031669 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:15.032807 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:15.084766 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:15.171936 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:15.183115 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:15.202221 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:15.210358 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:15.227254 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:20:15.233294 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:15.257248 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:15.266877 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:15.278967 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:15.301901 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:15.302649 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:15.417979 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:15.428731 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:15.441269 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:15.513271 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:15.514135 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:15.516920 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:15.524628 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:15.526034 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:15.542819 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:15.574772 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:15.586613 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:20:15.589375 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 14:20:15.607144 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:15.619560 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:15.635712 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:15.635886 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:15.728178 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:15.813633 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:15.926366 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:20:15.927910 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:15.937182 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:15.944159 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:15.946774 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:15.950552 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:15.954517 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:15.981021 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:16.010613 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:16.036944 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:16.040511 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:16.074530 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:16.084256 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:16.118423 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:16.123006 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:16.127848 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:16.134009 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:16.151469 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:16.151949 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:16.152709 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:16.153254 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:16.207427 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:20:16.297796 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:16.317340 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:16.349419 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:16.367084 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:16.376348 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:16.398355 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:16.399025 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:16.434420 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:16.439913 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:16.452582 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:16.454582 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:16.458755 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:16.507473 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:16.610699 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:16.623163 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:16.625145 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:16.630813 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:16.656006 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:16.656821 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:16.830971 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:16.846743 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:16.863493 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:16.869379 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:16.876032 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:16.887234 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:16.897758 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 14:20:16.897899 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:16.903061 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:16.921947 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:16.980051 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:16.989138 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:17.012754 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.032800 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.044751 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:17.052973 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:17.081833 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:17.083719 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:17.101680 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:17.102389 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:17.108605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:17.131960 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:17.202190 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:17.206624 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:17.223995 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:17.246397 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:17.265301 13470 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:17.265487 13470 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:17.266335 13470 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:20:17.269079 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:17.269187 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:17.269260 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:17.269557 13470 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:17.271536 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.309271 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 14:20:17.311023 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:17.329668 13470 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:17.329926 13470 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:17.330818 13470 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:20:17.335091 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:17.339931 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:17.341614 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:17.341980 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:20:17.342214 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
I20240503 14:20:17.347283 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:17.379349 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:17.392316 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:17.396356 2237 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.99.8:45929, user_credentials={real_user=slave}} blocked reactor thread for 61265.3us
W20240503 14:20:17.396374 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:17.402809 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:20:17.403050 2236 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.99.5:41737, user_credentials={real_user=slave}} blocked reactor thread for 62757.4us
I20240503 14:20:17.403110 13470 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:17.403671 2234 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 60775.5us
W20240503 14:20:17.443699 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.497990 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:20:17.506326 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.522578 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.562288 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:17.598744 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:17.605880 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:17.612102 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:17.616806 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:17.636234 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:17.636240 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:17.636636 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:17.736678 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:17.793926 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:17.830098 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:17.874637 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:17.904713 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:17.980329 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:17.990089 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:18.013427 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:18.040174 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:18.050489 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:18.059405 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:18.067447 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:18.079385 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:18.079784 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:18.085198 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:18.111064 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:18.111341 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:18.113270 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:18.116214 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:18.282655 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:18.286029 13628 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:18.286190 13628 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:18.286893 13628 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:18.291410 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:18.291844 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:18.291940 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:18.293073 13628 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:20:18.293251 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:18.294461 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:18.296352 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:18.308250 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:18.335857 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:18.405252 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:18.459525 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:18.508440 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:18.516719 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:18.528789 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:18.533769 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:18.540912 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:18.541625 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:18.541733 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:18.556134 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:18.559378 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:18.605999 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:18.639529 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:18.677261 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 14:20:18.714978 13827 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:18.715159 13827 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:18.715853 13827 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:18.717733 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:18.717885 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:18.718005 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:18.718470 13827 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:18.734745 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240503 14:20:18.781656 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:18.794601 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:18.795118 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:18.795603 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:18.804054 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:18.806716 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:18.926123 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:18.937084 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:18.939162 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:18.945129 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:18.954077 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:18.958417 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:18.975579 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:19.019340 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:19.027102 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:19.028755 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:19.028890 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:19.028950 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:19.031929 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:19.032783 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:19.034844 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:19.110888 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:19.225687 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:19.233103 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:19.263167 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:19.265225 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:19.273862 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:19.294976 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:19.350656 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:19.388962 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:19.419883 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:19.487717 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:19.496510 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:19.496634 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:19.505879 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:19.507694 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:19.521672 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:19.529086 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:19.530323 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:19.565718 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:19.598274 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:19.603018 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:19.608197 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 14:20:19.637923 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:19.642374 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:19.710222 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:20:19.742955 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:19.743557 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:19.764478 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:19.793591 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:19.795297 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:19.849437 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:19.864212 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:19.868767 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:19.929581 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:19.975006 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:20.017601 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:20.044133 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:20.105551 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:20.196062 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:20.196678 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:20.212723 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:20.220081 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:20.231441 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:20.241068 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:20.242857 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:20.289907 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:20.332317 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:20.408846 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:20.425727 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:20.428694 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:20.437016 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:20.442838 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:20.463016 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:20.464733 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:20.469344 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:20.470517 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:20.472440 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:20.473510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:20.547698 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:20.564230 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:20.642346 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 14:20:20.643491 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:20.666741 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:20.681982 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:20.682176 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:20.698948 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:20.699826 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:20.725068 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:20.773277 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:20.827699 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:20.921583 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:20.922236 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:20.934864 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:20.937860 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:20.955149 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:20:20.956154 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:20.956498 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:20.978701 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:20.994089 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:21.060869 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:21.063010 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:21.131047 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:21.143972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:21.148495 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:21.154901 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:21.161983 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:21.174302 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:21.184321 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:21.235997 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:21.251535 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:20:21.343163 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:20:21.377816 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:21.382056 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:21.411648 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:20:21.413820 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:21.428956 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:21.435081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:21.447165 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:21.451531 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:21.474649 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:21.477243 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:21.483510 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:21.500180 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:21.582607 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:21.598419 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:21.603907 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:21.620101 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:21.626698 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:21.627727 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:21.627900 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:21.639413 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:21.647512 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:21.834833 14307 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:21.835024 14307 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:21.835763 14307 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:21.837554 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:21.838675 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:21.842588 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 14:20:21.842823 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:21.842911 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:21.843159 14307 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:21.844964 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:21.854220 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 14:20:21.904970 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:21.920451 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:21.923955 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:21.931465 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:21.935158 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:21.944424 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:21.968674 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:22.002578 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:22.018419 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:22.033442 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:22.081604 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:22.150214 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:22.233588 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:22.317035 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:22.356469 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:22.359057 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:22.366901 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:22.387768 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:22.394912 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:22.395882 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:22.406124 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:22.413076 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:22.417729 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:22.456600 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:22.480363 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:22.517426 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:22.533932 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:22.565145 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:20:22.600414 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:22.605353 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:22.608196 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:22.609935 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:22.614485 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:22.654963 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:22.655683 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:22.799860 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:22.808892 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:22.830572 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:22.843628 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:22.866571 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:22.866971 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:22.878491 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:22.980016 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:22.980051 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:22.982885 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:23.003036 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:23.009200 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:23.059368 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:23.062260 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:23.064533 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:23.065634 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:23.067974 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:23.080760 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:23.081995 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:23.087038 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:23.090785 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:23.100149 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:23.112207 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:23.291031 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:23.297493 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:23.319820 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:23.331559 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:23.340410 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:23.357710 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:23.360738 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:23.389961 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:23.425411 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:23.432426 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:23.442416 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:23.502871 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:23.543963 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:23.546573 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:23.564079 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:23.570766 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:23.571328 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:23.577149 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:23.618683 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:23.801481 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:23.821873 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:23.822335 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:23.828811 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:23.829535 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:23.835871 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:23.839942 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:23.895478 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 14:20:23.899250 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:23.936964 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:23.966054 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:23.974936 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:24.026897 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 14:20:24.041280 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:24.042630 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:24.057679 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:24.062053 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:24.070284 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:24.077160 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:24.077565 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:24.111117 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:24.116585 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:24.193406 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:24.323679 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:24.345988 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:24.400035 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:24.448076 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:24.448695 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:24.470005 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:24.470237 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:24.488005 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:24.496718 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:24.508106 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:24.523437 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:24.524972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:24.541026 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:24.575749 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:24.608744 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 14:20:24.627112 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:24.707391 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:24.724264 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:24.734508 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:24.748118 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:24.756942 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:24.757249 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:24.777608 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:24.847172 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:24.848098 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:24.849426 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:24.898512 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:24.911727 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:24.935488 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:24.959419 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:24.981681 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:24.981760 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:24.982280 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:24.983856 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:24.992102 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:25.005048 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:25.007066 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:25.146031 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:25.186200 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:25.193559 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:25.195273 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:25.208523 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:25.211367 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:25.219671 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:25.247027 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:25.270040 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:25.346172 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:25.382731 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:25.391620 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:25.408623 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:25.426108 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:25.434453 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:25.448211 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:25.470978 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:25.561769 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:25.623422 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:25.633626 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:25.638720 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:25.666067 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:25.669023 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:25.696354 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:25.699745 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:25.722064 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:25.722298 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:25.724499 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:25.725595 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:25.756815 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:25.768761 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 14:20:25.786453 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:25.848309 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:25.857762 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:25.882696 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:25.899521 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:25.911752 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:25.962977 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:25.964972 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:25.965039 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:25.998900 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:26.024282 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:26.126253 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:20:26.138414 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:26.175966 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:26.192487 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:26.197247 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:26.215540 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:26.222214 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:26.222695 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:26.225004 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:26.297575 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:26.375505 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:20:26.385725 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:26.420466 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:26.436090 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:26.470224 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:26.590250 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:26.600210 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:26.642329 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:26.661078 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:26.663152 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:26.667018 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:26.672777 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:26.695991 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:26.709733 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 14:20:26.729048 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:26.774989 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:26.783882 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:26.809609 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:26.841049 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:26.865007 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:26.872879 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:26.888922 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:26.904477 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:26.910552 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:26.918455 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:26.920172 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:26.925052 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:26.931886 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:26.977465 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:26.992167 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:27.110699 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:27.113731 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:27.118664 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:27.129828 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 14:20:27.135746 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:27.137127 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:27.139333 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:27.142051 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:27.150873 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:27.177428 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:27.194929 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:27.335264 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:27.396654 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:27.413398 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:27.423624 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:27.429555 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:27.452448 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:27.473374 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:27.502408 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:27.504143 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:27.520318 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:27.552564 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:27.575114 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:27.583269 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 14:20:27.638897 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:27.663368 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:27.678117 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:27.678985 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:27.683432 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:27.703405 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:27.729202 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:27.732514 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:27.785429 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:27.808043 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:27.905239 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:27.916615 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:27.929667 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:27.930832 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:27.941591 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:27.982549 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:28.004439 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:28.070714 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:28.090370 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:28.099870 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:28.148015 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:28.156128 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:28.162350 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:28.164553 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:28.168043 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:28.168711 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:28.200405 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:28.202369 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:28.229190 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:28.238075 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:28.243054 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:28.244169 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:28.269109 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:28.330754 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:28.371295 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:28.402120 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:28.408906 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:28.411847 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:28.412573 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:28.436733 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:28.569046 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:28.589427 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:28.644469 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:28.802212 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:28.808231 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:28.827221 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:28.828011 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:28.831161 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:28.840385 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:28.853786 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:28.854709 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:28.856845 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:28.903532 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:28.977718 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:29.029419 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:29.031903 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:29.038703 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:29.076453 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:29.081830 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:29.087517 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 14:20:29.094740 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:29.099644 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:29.102910 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:29.167137 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:29.202244 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:29.282409 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:29.323153 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:29.324246 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:29.335141 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:29.348078 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:29.354763 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:29.364189 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:29.398129 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:29.415625 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:29.419898 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:29.431243 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:29.438741 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:29.451347 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:29.453665 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:29.457497 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:29.575999 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:29.582878 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:29.635103 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:29.649717 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:29.672132 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:29.674991 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:29.678619 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:29.690057 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:29.724681 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 14:20:29.793519 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:29.897444 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:29.898931 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:29.922538 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:29.946810 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:29.947535 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:29.949144 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:29.954211 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:29.963090 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:29.985101 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:30.108708 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:30.112730 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:20:30.131740 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:30.133124 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:30.143631 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:30.151485 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:30.163906 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:30.165793 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:30.185050 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:30.188987 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:30.244886 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:30.292631 15436 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:30.292838 15436 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:30.293846 15436 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:20:30.294174 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:20:30.294384 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:20:30.296252 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:30.296478 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:30.296609 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:30.297101 15436 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:20:30.367094 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:30.370020 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:30.397814 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:30.401543 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:30.403081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:30.429258 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:20:30.436928 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:30.447347 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:30.470199 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:30.489341 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:30.501969 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:30.535578 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:30.535774 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:30.559366 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:30.634162 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:30.640904 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:30.658838 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:30.671298 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:30.674420 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:30.690461 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:30.693928 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:30.712852 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:30.785109 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:30.794438 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:20:30.807241 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:30.835050 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:30.901459 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:30.905794 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:30.925343 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:30.989004 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:31.075556 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:31.075547 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:31.076743 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:31.080056 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:31.089340 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:31.093565 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:31.126542 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:20:31.128044 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:31.128471 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:31.286547 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:31.294732 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:31.294835 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:31.295246 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:31.318554 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:31.327168 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:31.403358 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:31.435014 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:20:31.437654 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:31.472509 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:31.475392 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:31.506412 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:31.548810 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:31.550837 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:31.568930 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:31.569315 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:31.581190 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:31.608908 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:31.631250 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:31.675988 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:31.697701 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:31.707245 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:20:31.765123 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:31.786543 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:31.789090 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:31.793308 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:31.794476 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:31.796732 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:31.814261 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:31.848407 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:31.869009 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:31.886622 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:31.890066 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:31.960425 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:31.995683 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:32.014735 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 14:20:32.016683 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:32.017079 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:32.038360 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:32.040463 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:32.043651 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:32.077031 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:32.114006 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:32.132287 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 14:20:32.216419 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:32.226291 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:32.234947 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:32.239630 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:32.242404 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:32.261139 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:32.277098 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:32.280220 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:32.340188 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:32.378424 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:32.416921 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:32.420608 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:32.493630 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:32.516541 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:32.517237 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:32.519013 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:32.524209 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:32.541658 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:32.591883 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:32.682932 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:32.691663 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:32.734320 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:32.749610 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:32.751463 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:32.759459 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:32.782500 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:32.798509 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:32.798831 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:32.805022 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:20:32.810290 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:32.819655 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:32.874751 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:32.965975 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:32.990568 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:33.015419 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:33.018096 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:33.035013 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:33.156702 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:33.184913 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:33.188872 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:20:33.192993 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:33.209167 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:33.209686 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:33.243861 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:33.345966 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:33.366488 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:33.379406 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:33.385326 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:33.386405 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:33.399996 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:33.437582 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:33.440233 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:33.449534 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:33.461289 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:33.465533 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:33.475669 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:33.486850 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:33.493436 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:33.515379 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:33.667673 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:33.673612 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:33.689934 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:33.700371 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:33.705224 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:33.709535 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:33.712399 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:33.783684 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:33.927258 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:33.933808 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:33.937891 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:33.951479 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:33.952335 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:33.970428 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:33.987886 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:33.994369 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:33.994581 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:33.998131 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:34.039268 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:34.116149 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240503 14:20:34.162019 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:34.167470 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:34.171568 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:34.173805 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 14:20:34.174304 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:34.175272 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:34.265832 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:34.309006 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:34.349368 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:34.385505 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:34.392014 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:34.413308 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:34.415071 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:34.423170 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:34.434128 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:34.435020 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:34.437422 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:34.454972 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:34.457310 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:34.476924 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:34.506783 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:34.535681 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:34.541149 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:34.582731 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:34.590727 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 14:20:34.640686 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:34.652254 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:34.662149 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:34.688722 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:34.695955 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:34.707404 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:34.715553 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:34.717609 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:34.720153 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:34.732358 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:34.901540 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:34.903760 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:34.924021 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:34.932965 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:34.936560 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:34.963876 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:35.012562 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:35.013157 15579 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:35.013319 15579 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:35.014061 15579 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:20:35.015733 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:35.016078 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:35.016182 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
W20240503 14:20:35.016319 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:35.016422 15579 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:35.018020 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:35.047394 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:35.126368 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:35.127301 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:35.192852 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:35.290594 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:35.305437 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:35.349542 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:35.357583 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:35.375926 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:35.392925 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:35.396571 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:35.400238 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:35.417837 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:35.495847 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:35.519130 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:35.534902 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:35.580015 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:35.594590 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:35.608362 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:35.609076 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:35.613379 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:35.615134 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:35.620237 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:35.624012 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:35.632998 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 14:20:35.638368 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:35.814368 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:35.817314 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240503 14:20:35.825690 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:35.827343 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:35.833047 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:35.839242 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:35.846879 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:35.863503 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:35.873476 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:35.882440 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:35.954229 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:36.008342 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:36.026391 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:36.043895 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:36.053136 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:36.062389 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:36.063783 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:36.086663 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:36.094472 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:36.094580 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:36.144482 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:20:36.158969 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:36.287786 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:36.293308 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:36.300724 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:36.305419 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:36.311522 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:36.316707 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:36.359376 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:36.409181 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:36.428062 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:36.439950 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:36.449780 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:36.514135 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:36.517390 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:36.535022 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:36.545989 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:36.565253 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:36.565675 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:36.569518 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:36.585659 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240503 14:20:36.746362 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:36.763918 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:36.768759 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:36.802055 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:36.810034 16457 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:36.810218 16457 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:36.810973 16457 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:20:36.814347 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:36.814332 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:36.818133 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:36.818648 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:36.818764 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:36.819110 16457 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:20:36.823663 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:36.835470 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:36.853951 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:36.858837 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:36.863248 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:36.872944 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:36.872953 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:36.913362 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:36.964208 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:36.986908 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:37.001019 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:37.050673 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:37.051127 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:37.058776 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:37.074834 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:37.081825 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:37.084250 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:37.139914 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:37.159152 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:37.206624 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:37.220408 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:37.221422 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:37.226181 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 14:20:37.392920 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:37.447580 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:37.466341 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:37.472543 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:37.489867 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:37.490868 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:37.491372 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:37.499135 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:37.507586 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:37.530956 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:37.552405 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:37.573333 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:37.696820 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:37.708025 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:37.716782 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:37.729198 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:37.738749 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:37.746955 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:37.753931 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:37.784886 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:37.822134 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:20:37.875816 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:37.924782 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:37.930852 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:37.945693 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:37.960078 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:37.962356 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:37.966394 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:37.991219 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:37.992302 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:38.036921 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:38.143662 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:38.148327 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 14:20:38.176836 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:38.189785 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:38.198351 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:38.200093 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:38.208832 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:38.228570 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:38.239531 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:38.292654 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:38.360821 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:38.361243 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:38.395845 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:38.404554 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 14:20:38.427839 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:38.428959 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:38.445979 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:38.446249 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:38.463294 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:38.468981 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:38.471275 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:38.485255 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:38.510792 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:38.531864 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:38.655277 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:38.657009 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:38.669875 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:38.671216 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:38.676152 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:38.740656 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:38.754323 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:38.813047 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:38.819007 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:38.880582 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:38.905020 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:38.936904 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:38.955770 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:38.972849 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:38.978698 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:38.992838 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:39.036541 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:39.079702 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:39.106168 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:39.118664 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:39.121183 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:39.161911 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:39.169271 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:39.179790 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:39.206321 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:39.225852 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:39.241282 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:20:39.248729 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:39.268770 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:39.284543 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:39.341039 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:39.368624 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:39.379280 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:39.426051 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:39.442714 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:39.460278 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:39.465416 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 14:20:39.507925 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:39.548298 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:39.561681 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:39.592895 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:39.612146 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:39.625620 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:39.627452 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:39.629237 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:39.635143 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:39.636646 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:39.689206 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:39.693513 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:39.748162 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:39.823428 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:39.839761 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:39.839874 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:39.843744 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:39.844112 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:39.861575 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:39.990180 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:40.029783 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:40.061689 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:40.066183 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:40.073094 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:40.083338 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:40.084090 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:40.088836 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:40.102628 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:40.108059 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:40.122092 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:40.175096 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:40.179500 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:40.281045 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:40.307006 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:40.318387 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:40.334509 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:40.335968 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:40.336397 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:40.340216 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:40.493773 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:40.529393 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:40.554373 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:40.577090 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:40.592481 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:40.597268 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:40.601732 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:40.613698 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:40.619923 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:40.623311 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:40.634132 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:40.711433 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:40.792342 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:40.801899 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:40.817293 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:40.825793 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:40.829659 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:40.832721 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:40.849973 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 14:20:40.856657 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:40.907667 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:40.925176 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:40.951778 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:40.961763 17014 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:40.961962 17014 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:40.962707 17014 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:40.964046 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:40.964532 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:40.964648 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:40.964907 17014 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:40.966190 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:41.005456 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.008435 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:41.012212 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:41.035629 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:41.049199 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:41.079360 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:41.100773 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:41.101861 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:41.108795 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:41.114742 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:41.154278 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:41.202708 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:20:41.209121 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:41.254369 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.316121 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:41.324167 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:41.328740 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:41.340226 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:41.348265 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:41.355347 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:41.381129 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:41.387259 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.504871 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.509882 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:41.518005 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:41.593194 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:41.629277 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.654517 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:20:41.665586 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:41.763924 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:41.779934 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:41.783821 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:41.786850 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:41.789503 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:41.794929 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:41.795918 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:41.815668 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:41.860605 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:41.870112 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:41.914129 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:41.960651 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:41.976891 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.987701 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:41.989782 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:42.007519 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:42.015710 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:42.020215 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:42.038024 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:42.039662 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:42.050658 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:42.051658 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:42.051786 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:42.074519 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:42.122418 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 14:20:42.164544 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:42.174129 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:42.249636 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:42.267666 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:42.275339 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:42.279347 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:42.294855 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:42.323777 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:42.371292 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:42.425072 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:42.439777 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:42.449184 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:42.534452 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:42.537585 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:42.543463 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:42.562036 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:42.564608 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:42.575446 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:42.580452 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:42.630770 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:42.708415 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:42.771507 16576 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:42.771754 16576 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:42.773000 16576 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:20:42.775359 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
W20240503 14:20:42.775604 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:42.775667 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:20:42.781905 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:42.782590 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:42.782831 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:42.783478 16576 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:20:42.789698 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:42.794852 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:42.807644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:42.813871 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:42.819298 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:42.868984 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:42.873219 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:42.904662 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:20:42.953840 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:42.981066 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:42.984979 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:42.992789 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:43.019493 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:43.027940 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:43.028192 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:43.049134 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:43.073243 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:43.080861 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:43.128859 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:43.219506 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:43.232749 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:43.233747 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:43.239800 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:43.251430 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:43.255398 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:43.256280 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:43.405997 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:43.434309 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:43.444515 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:43.481770 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:43.495098 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:43.499271 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:43.502647 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:43.503767 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:43.506446 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:43.511468 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:43.555876 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:43.561921 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:43.573899 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:43.598682 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:43.651711 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:43.653993 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:43.690985 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:43.692786 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:43.736308 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:43.798486 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:43.873956 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:43.936420 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:43.957121 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:43.959288 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:43.959343 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:43.965842 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:43.974014 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:43.999130 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:44.068145 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:44.133615 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:20:44.145051 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:44.150839 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:44.157647 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:44.158788 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:44.167359 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:44.179450 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:44.180415 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:44.181063 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:44.199709 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:44.209106 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:44.244836 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:44.255254 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:44.313565 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:44.314965 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:44.408677 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:44.415892 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:44.440703 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:20:44.441532 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 14:20:44.443346 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:44.444221 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:44.463478 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:44.471441 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:44.499843 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:44.529057 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:44.529392 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:44.563748 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 14:20:44.644043 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:44.650465 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:44.651684 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:44.665812 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:44.673700 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:44.676522 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:44.683226 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:44.690781 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:44.701151 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:44.735443 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:44.737860 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:44.812505 17631 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:44.812731 17631 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:44.813374 17631 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:44.814639 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:44.815094 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:44.815222 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:44.815464 17631 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:44.818056 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:44.873131 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:44.884457 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:44.885442 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:44.912252 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:44.923755 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:45.063539 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:45.113195 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:45.121768 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:45.127295 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:45.129820 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:45.142716 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:45.143626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:45.158339 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:45.243206 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:45.271834 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:45.275944 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:45.288338 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:45.334497 17518 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:45.334738 17518 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:45.335778 17518 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:20:45.339644 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:20:45.339956 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:20:45.340270 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:45.342784 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:45.342959 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:45.343318 17518 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:20:45.370288 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:45.374329 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:45.382411 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:45.397855 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:45.404484 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:45.419641 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:45.447705 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:45.509374 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:45.535426 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:45.642948 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:45.650153 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:45.652835 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:45.670828 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:45.671373 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:45.679479 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:45.732910 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:45.744905 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:45.747532 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:45.760388 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 14:20:45.772284 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:45.843742 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:45.887367 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 14:20:45.888372 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:45.996368 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:46.025156 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:46.031335 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:46.032224 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:20:46.070619 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:46.080395 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:46.101929 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:46.110020 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:46.125263 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:46.134930 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:46.151027 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:46.152453 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:46.167420 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:20:46.179026 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:46.239353 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:46.261166 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 14:20:46.325045 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:46.349999 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:46.354724 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:46.364941 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:20:46.369727 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:46.374060 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:46.374991 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:46.400456 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:46.430150 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:46.439117 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:46.467324 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:46.481779 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:46.600670 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:46.611316 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:46.620348 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:46.621028 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:46.630414 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:46.638732 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:46.687857 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:46.701344 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:46.715847 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:46.716542 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240503 14:20:46.717578 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:46.769733 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:46.776280 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:46.776855 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:46.785459 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:20:46.851600 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:46.857460 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:46.866044 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:46.873626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:46.874749 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:46.915959 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:46.922744 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:46.925560 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:47.026777 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:47.050534 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:47.055653 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 14:20:47.080228 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:47.086095 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:47.105564 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:47.105720 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:47.106407 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:47.185402 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:47.260964 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:47.295449 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:47.298444 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:47.300122 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:47.314009 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:47.315403 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:47.318324 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:47.324776 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:47.425477 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:47.477301 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:47.491220 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:47.517560 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:47.540423 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:47.544836 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:47.548341 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:47.580890 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:47.587272 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:47.596800 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:47.648144 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:47.724412 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:47.753270 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:47.786669 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:47.792109 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:47.793133 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:47.795241 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:47.824822 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:47.829241 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:47.886986 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:47.952292 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:47.956326 17872 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:47.956473 17872 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:47.957077 17872 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:47.958221 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:47.958437 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:47.958521 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:47.958698 17872 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:47.981863 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:48.008349 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:48.067078 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:20:48.067991 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:48.216454 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:48.250871 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:48.254500 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:48.275260 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:48.291795 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:48.294188 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:48.317057 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:48.321631 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:48.351012 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:48.368292 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 14:20:48.373425 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:48.489269 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:48.533681 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:48.537487 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:48.538713 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:48.548439 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:48.549592 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:48.557107 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:48.573926 4268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.3:45269, user_credentials={real_user=slave}} blocked reactor thread for 54571.9us
I20240503 14:20:48.590164 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:48.608783 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:48.609848 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:48.621572 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:48.628656 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:48.698366 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:48.734438 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:48.735085 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:48.756563 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:48.767833 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:48.800740 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:48.801455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:48.806725 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:48.809919 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:48.812417 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:48.830811 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:48.857659 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:48.859738 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:48.874328 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:48.902338 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:48.902707 17872 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:48.902889 17872 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:20:48.903558 17872 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:20:48.904851 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:48.905309 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:48.905411 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:48.905654 17872 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:48.924243 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:48.929455 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:48.950213 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:49.024140 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:49.028126 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:49.030267 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:49.037434 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:49.041672 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:49.141839 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:49.164119 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:49.183135 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:49.232719 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:49.232831 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:49.249893 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:49.264689 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:49.284130 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:49.296149 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:49.318624 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:49.345329 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:49.382696 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:49.460703 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:20:49.460857 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:49.463299 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 14:20:49.469795 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:49.480340 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:49.499023 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:49.502326 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:49.505168 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:49.509981 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:49.512701 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:49.540628 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:49.603161 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:49.637501 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:49.711704 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:49.716684 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:49.743201 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:49.749835 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:49.758111 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:49.764115 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:49.769389 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:49.781239 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:49.833712 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:49.925549 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:49.936760 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:49.964615 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:49.967259 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:50.005071 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:50.015101 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:50.021817 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:50.031405 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:50.032166 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:50.033387 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:50.040903 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:50.245739 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:50.276208 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:50.293155 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:50.331993 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:50.360795 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:50.379720 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:50.417935 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:50.435424 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:50.442055 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:50.443989 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:50.446879 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:50.449064 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:50.470595 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:50.639083 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:50.642572 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:50.656981 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:50.666738 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:50.685439 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:50.687654 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:50.707603 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:50.754812 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:50.758282 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:50.765656 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:50.774245 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:50.824062 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:50.872622 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:50.894068 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:50.896683 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:50.904660 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:50.906545 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:50.930440 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:50.932261 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:50.943317 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 128 similar messages]
I20240503 14:20:50.953691 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:50.987591 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:51.056452 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:51.114967 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:51.123478 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:20:51.127055 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:51.133206 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:51.139845 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:51.140030 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:51.175334 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:51.180040 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:51.234634 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.237224 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:51.262532 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 105 similar messages]
W20240503 14:20:51.263489 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:51.297696 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.304895 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.307332 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:51.315140 18442 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:20:51.315380 18442 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:51.316269 18442 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:20:51.318703 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.319550 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:51.319701 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:20:51.319985 18442 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:51.322181 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:51.324910 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:51.373260 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:51.393877 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:51.408987 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:51.409143 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:51.424988 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:51.425657 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:51.441177 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:51.490034 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 14:20:51.571380 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:20:51.594049 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:51.610105 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:51.626580 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:51.628232 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:51.634874 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:51.649048 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:51.676143 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.683226 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.703428 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:51.748930 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:51.787751 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:51.840943 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:51.846818 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:20:51.848186 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:51.848851 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:51.864598 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:51.867780 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:51.904237 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:51.995440 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:52.013859 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:52.053663 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:52.077682 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:20:52.083112 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:52.088119 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:52.102210 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:52.103621 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:52.110869 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 14:20:52.115444 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:52.138092 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:52.152211 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:52.158232 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:52.258044 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:52.274628 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:52.276736 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:52.319265 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:52.412364 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:52.429569 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:52.512724 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:52.522207 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:52.537797 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:52.542136 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:52.546136 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:52.564837 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:52.567382 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:52.621274 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:52.695600 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:52.708011 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:52.725381 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:52.741425 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:52.762305 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:52.767145 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:52.777433 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:52.779026 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:52.798261 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:52.895247 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:52.906118 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:52.916311 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:52.929065 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:20:52.995121 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:53.004400 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:53.030952 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:53.034189 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:53.035135 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:53.035218 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:53.070139 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:53.135025 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:20:53.176839 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:53.232141 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:53.245324 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:53.259780 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:53.266070 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:53.284158 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:20:53.297932 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:53.298847 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:53.310041 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:53.310726 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:53.320257 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:53.322330 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:53.378650 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 14:20:53.495667 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:53.504612 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:53.514842 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:53.523916 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:53.524849 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:53.556967 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:53.590401 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:53.630307 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:53.662950 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:53.686755 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:53.725126 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:53.725481 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:53.748984 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:20:53.762598 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:53.765549 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:53.767581 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:53.777441 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:53.779044 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:53.780412 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:53.793637 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:53.800237 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:53.853986 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:53.980546 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:53.993414 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:54.008751 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:54.019975 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:54.033938 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:20:54.041600 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:54.058061 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:54.105201 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:54.107686 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:20:54.149269 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:54.202744 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:54.218569 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:54.221187 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:20:54.242059 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:54.243746 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:20:54.262346 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:54.267549 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:54.270913 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:54.285027 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:54.289386 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:20:54.308169 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:54.315917 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:54.516647 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:20:54.550395 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 14:20:54.566371 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:54.627094 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:54.647724 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:54.652387 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:54.667793 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:54.680426 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:54.681381 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:54.685551 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:54.694123 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:54.719612 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:54.743899 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:54.796465 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:54.823031 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:54.841532 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:54.857050 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:54.872735 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:54.894173 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:54.911738 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:54.912127 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:54.913774 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:54.977557 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:55.017930 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:55.040517 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:55.077739 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:55.134562 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:20:55.147553 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:55.148517 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:55.162214 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:55.172636 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:55.179994 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:55.188962 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:55.212508 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:55.236937 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:55.374734 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:55.380759 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:55.430114 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 14:20:55.437845 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:55.445001 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:55.448748 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:55.450124 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:55.453776 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:55.468528 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:55.495841 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:55.544360 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:55.579216 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:55.633050 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:55.643134 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:55.655740 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:20:55.682801 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:55.684775 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:55.694346 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:55.702042 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:55.705736 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:55.785936 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:55.901593 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:55.907263 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:55.908331 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:20:55.926029 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:55.942459 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:55.945053 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:55.953313 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:55.958949 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240503 14:20:55.976971 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:55.977142 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:55.996726 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:55.999426 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:56.054313 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:56.105213 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:56.121840 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:56.128141 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:56.147037 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:20:56.155516 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:20:56.180670 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:56.180819 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:56.195814 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:56.196287 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:56.210369 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:56.229838 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:56.253068 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:56.278338 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:20:56.288122 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:56.402211 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:56.429193 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:56.439241 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:56.440286 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:56.445451 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:56.471128 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:56.498396 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:56.509815 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:56.516335 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:20:56.530444 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:56.625108 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:56.625314 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:56.635414 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:56.638864 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:20:56.704748 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:56.858433 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:20:56.859927 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:56.875448 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:56.893837 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:56.898877 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:56.902324 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:56.938995 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:56.961866 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:20:56.974299 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 14:20:56.979444 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:56.991261 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:57.013010 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:57.017118 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:57.080708 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:57.099893 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:57.100401 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:57.107926 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:57.112861 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:57.136710 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:57.147850 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:57.196555 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:57.211419 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:57.270838 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:57.332700 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:57.336278 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:57.359272 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:57.360905 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:57.360924 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:57.375464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:57.405267 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:57.415388 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:57.420684 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:57.431560 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:57.465953 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:57.528430 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:57.537016 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:57.621954 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:57.624222 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:57.643543 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:57.675716 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:57.681602 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:57.745806 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:57.756950 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:57.811432 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:57.861912 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:57.889955 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:57.893747 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:57.899442 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:57.905632 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:57.909852 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:57.936684 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:57.983407 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:58.036149 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:58.094321 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:58.106586 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:58.122918 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:58.135968 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:20:58.136893 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:20:58.140369 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 14:20:58.148525 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:58.162899 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:58.168735 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 14:20:58.180467 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:20:58.236435 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:58.243000 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:58.345248 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:58.370097 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:58.398916 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:58.402688 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:58.415429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:58.421324 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:58.432539 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:20:58.433256 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:58.478708 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:58.530750 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:58.545341 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:58.563022 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:58.590902 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:20:58.603497 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:58.614413 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:58.626607 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:58.637003 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:58.643471 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:58.669132 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:58.669371 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:20:58.683749 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:58.686667 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:58.729729 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:20:58.757658 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:58.803046 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:20:58.847949 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:58.907598 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:59.004789 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:20:59.035686 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:59.040656 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:20:59.043854 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:59.055930 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:59.056844 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:59.056847 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:20:59.058164 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:59.064965 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:20:59.128562 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:20:59.152299 19550 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
W20240503 14:20:59.152670 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:59.152530 19550 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:59.153594 19550 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:20:59.155666 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:59.156114 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:59.156208 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:59.156443 19550 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:59.159016 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:59.206451 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:20:59.297204 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:59.300025 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:20:59.303241 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:20:59.305431 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:59.305606 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:59.324025 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:59.328570 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:59.413578 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 14:20:59.442189 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:59.520767 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:20:59.556804 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:59.558204 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:59.560899 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:59.564867 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:59.573377 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:20:59.578642 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:20:59.612723 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:20:59.618628 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:20:59.628672 19550 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:20:59.628902 19550 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:20:59.629799 19550 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:20:59.630599 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:20:59.630848 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:20:59.631290 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:59.631937 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:20:59.632045 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:20:59.632313 19550 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:20:59.664002 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:20:59.713538 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:20:59.763689 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:59.765812 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:20:59.767573 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:20:59.783424 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:20:59.790163 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:20:59.791960 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:20:59.798738 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:20:59.807972 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:20:59.862445 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:59.931447 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:20:59.932389 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:20:59.949399 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:00.012352 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:00.018810 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:00.024137 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 14:21:00.025535 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:00.040287 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:00.041266 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:00.042970 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:00.066586 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:00.137939 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:00.218538 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:00.230403 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:00.232187 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:00.249936 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:00.254630 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:00.259794 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:00.266016 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:00.336536 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:00.423662 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:21:00.427521 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:00.453557 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:00.467614 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:00.490063 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:00.495597 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:00.508615 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:00.509590 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:00.510314 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:21:00.537166 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:00.609624 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:00.634266 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:21:00.691803 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:00.699065 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:00.711215 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:00.723871 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:00.727519 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:21:00.751457 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:00.765870 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:00.792255 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:00.797132 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:21:00.863323 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:00.896811 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:00.969825 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 14:21:01.020993 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:01.028183 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:01.066170 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:01.093874 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:01.137830 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:01.160696 19697 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:01.160868 19697 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:01.161573 19697 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:01.163645 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:01.164256 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:01.164419 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:01.165333 19697 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:01.169801 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:01.174376 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:01.179486 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:21:01.184446 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:01.188457 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:01.215647 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:01.242818 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:01.269809 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:01.292097 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:21:01.296923 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:01.304322 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:01.345041 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:01.367619 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:01.369345 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:21:01.425135 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:01.440816 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:01.451278 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:01.451618 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:01.471411 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:01.502573 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:01.563436 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:21:01.654646 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:01.661304 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:01.663314 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:01.667253 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:01.667299 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:01.672395 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:01.706554 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 14:21:01.739442 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:01.746102 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:01.750042 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:01.783702 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:01.795228 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:01.820600 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:01.845512 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:01.896802 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:01.899613 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:01.920879 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:01.938005 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:01.944239 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:01.966910 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:01.984326 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:02.015774 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:21:02.113729 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:02.150911 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:02.160821 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:02.163064 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:02.169302 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:02.174453 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:02.196359 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:02.215543 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:02.218500 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:02.287820 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:02.297494 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:02.308219 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:02.320730 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:02.362958 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:02.363512 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:02.377918 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:02.379244 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:02.393608 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:02.397516 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 14:21:02.417567 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:02.449390 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:02.474216 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:02.542485 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:02.609535 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:02.614943 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:02.629547 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:21:02.637336 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:02.638013 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:02.638089 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:02.643908 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:02.665283 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:02.702409 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:02.751065 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:02.836351 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:02.836805 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:02.878683 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:02.885455 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:02.886246 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:02.891896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:02.922833 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:02.944981 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:02.966307 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:03.057708 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:03.097254 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:03.126590 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:03.126681 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:03.162338 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:03.183434 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:21:03.208643 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:03.251328 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:03.288327 19550 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:03.288580 19550 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:03.289652 19550 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:03.290783 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:21:03.291006 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:21:03.291718 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:03.292362 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:03.292486 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:03.292742 19550 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:03.321059 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:03.322978 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:03.331739 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:03.339911 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:03.343449 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:03.345392 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:03.357224 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:03.358083 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:03.372689 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:03.544431 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:03.560268 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:21:03.564221 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:03.574549 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:03.574944 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:03.583552 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:03.612110 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:03.623473 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:03.680385 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:03.718725 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:03.733412 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:03.742106 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:03.763123 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:03.764983 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:03.807008 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:03.826920 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:03.832302 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:03.836745 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:03.840595 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:03.841476 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:03.904130 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:03.973572 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:04.005514 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:04.022898 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:04.042238 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:04.059680 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:04.063848 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:04.065146 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:04.068521 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:04.079367 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:04.094309 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:04.096693 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 14:21:04.130277 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:04.155877 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:04.202149 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:04.255395 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:04.282876 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:04.290799 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:04.303552 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:04.318717 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:04.334326 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:04.351868 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:04.361016 2495 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 45105.7us
I20240503 14:21:04.394518 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:04.413522 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:04.455219 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:04.457911 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:04.503541 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:04.568972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:04.570554 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:04.582566 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:04.590173 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:04.590463 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:04.609350 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:04.666728 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:04.727063 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:04.758349 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 14:21:04.817111 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:04.833995 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:04.847931 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:04.852491 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:21:04.852802 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:04.856639 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:04.866274 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:04.890390 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:04.930122 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:04.936028 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:04.942425 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:04.982188 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:05.051432 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:21:05.053933 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:05.059914 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:05.079684 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:05.093245 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:05.116010 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:05.121825 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:05.129952 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:05.189934 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:05.256435 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:05.363476 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:05.382899 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:05.413612 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:05.502995 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:05.506929 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:05.510208 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:05.513381 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:05.533497 19550 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:05.533752 19550 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:05.534684 19550 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:05.538046 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:21:05.538280 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:21:05.538916 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:05.539219 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:05.539333 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:05.539796 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:05.540555 19550 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:05.547027 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:05.565155 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:05.569710 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:05.593053 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:05.611668 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:05.655762 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:05.698202 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:21:05.749583 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:05.760190 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:05.769287 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:05.771613 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:05.772405 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:21:05.786554 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:05.789839 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:05.943470 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:05.970837 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240503 14:21:05.971920 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 14:21:06.011278 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:06.015152 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:06.021520 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:06.024389 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:06.027777 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:06.034915 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:06.057533 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:06.059589 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:06.088972 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:06.111460 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:06.121723 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:06.122152 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:06.125501 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:06.125939 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:06.200161 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:06.220376 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:06.227916 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:06.229146 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:06.247288 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:06.256691 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:06.364692 2497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:21:06.370263 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:06.415881 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:06.441601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:06.445344 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:06.454303 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:06.454315 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:06.470893 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:06.475983 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:06.491780 20426 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:06.491981 20426 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:06.492642 20426 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:06.498467 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:06.499601 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:06.499790 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:21:06.500036 20426 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:06.506008 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:06.509337 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:06.521286 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:06.576624 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 14:21:06.595679 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:06.634124 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:06.686548 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:06.691663 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:06.709460 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:06.718147 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:06.723415 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:06.728615 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:06.729264 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:06.731885 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:06.783878 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:06.805662 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:06.854887 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:06.904500 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:06.915833 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:06.933240 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:06.938987 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:06.953490 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:06.957306 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:06.976258 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:07.022693 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240503 14:21:07.061622 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:07.070842 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:07.145598 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:07.156930 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:07.171963 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:07.172415 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:07.173352 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:07.191275 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:21:07.233273 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:07.290589 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:07.291877 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:07.340399 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:07.377887 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:07.455204 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:07.472842 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:07.567503 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:07.596743 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:07.601342 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:07.601702 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:07.602766 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:07.604733 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:07.607944 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:07.616698 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:07.665731 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:07.670127 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:07.687480 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:07.796437 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:07.808986 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:07.816365 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:07.820866 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:07.840461 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:07.842411 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:07.854446 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:07.908648 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:07.936900 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:07.950906 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:07.968569 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:08.036247 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:08.048269 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:08.052996 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:08.060142 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:08.060376 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:08.066699 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:08.075439 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:08.097532 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:08.116828 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:08.145500 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:08.189451 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:08.205780 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:21:08.304250 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:08.306039 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:08.322485 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:08.324043 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:08.334226 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:08.337199 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:08.362828 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:08.397047 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:21:08.479910 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:08.483543 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 14:21:08.498000 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:08.570024 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:08.571597 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:08.581570 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:08.587102 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:08.588608 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:08.595130 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:08.596997 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:08.658514 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:08.722327 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:08.768239 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:08.773329 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:08.782331 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:08.786585 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:08.794106 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:08.795127 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:08.797468 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:08.811129 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:08.812188 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:08.915874 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:08.938788 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:08.976828 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:08.999032 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:09.020010 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:09.024637 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:09.031530 20763 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:09.031709 20763 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:09.032557 20763 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:09.035244 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:09.037710 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:09.039139 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:09.039268 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:09.039501 20763 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:09.040017 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:09.041420 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:09.060498 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:09.065456 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:09.118669 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:09.138836 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:09.146139 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:09.189504 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 14:21:09.247952 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:09.257719 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:09.268796 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:09.285233 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:09.289475 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:09.309283 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:09.326773 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:09.330904 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:09.336583 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:09.381736 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:09.407411 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:09.439044 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:09.446413 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:09.469245 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:09.709126 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:09.716567 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:09.724759 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:09.728101 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:09.734948 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:09.737087 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:09.746803 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:09.754026 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:09.783365 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:09.817935 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240503 14:21:09.824638 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:09.847342 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:09.856639 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:09.960638 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:21:09.967378 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:09.976065 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:09.980513 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:09.996910 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:09.997126 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:10.006903 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:10.033402 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:10.058349 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:10.080036 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:10.118993 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:10.180112 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 14:21:10.190291 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:10.197636 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:10.202404 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:10.218015 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:10.221477 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:10.238265 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:10.248329 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:10.325103 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:10.369884 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:10.409183 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:10.429559 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:10.431012 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:10.456926 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:10.470583 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:21:10.470994 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:10.501106 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:10.504009 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:10.505089 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:10.561973 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:21:10.714110 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:10.721566 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:10.736006 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:10.739503 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:10.751475 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:10.757948 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:10.759629 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:10.809561 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:10.813197 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:10.853348 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:10.930246 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:10.941998 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:10.943601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:10.948151 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:10.955832 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:10.960513 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:10.996668 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240503 14:21:10.998003 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:11.014137 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:11.023306 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:11.053959 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:11.062681 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:11.178121 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:11.192899 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:11.196961 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:11.199187 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:11.205201 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:11.219955 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:11.247105 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:11.253587 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:11.293865 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:11.349681 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:11.373875 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:11.387840 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 14:21:11.388892 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:11.412659 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:11.415458 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:11.421067 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:11.438709 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:11.438925 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:11.464555 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:11.471545 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:11.482800 21328 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:11.482995 21328 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:11.483876 21328 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:11.484917 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:21:11.485122 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:21:11.490828 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:11.491767 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:11.496455 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:11.496771 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:11.497182 21328 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:11.512001 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:11.625588 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 14:21:11.703612 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:11.742801 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:11.778795 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:11.809441 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:11.873276 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:11.887859 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:11.896306 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:11.905484 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:11.929352 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:11.929495 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:11.935498 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:11.940464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:11.974895 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:11.987434 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:11.988687 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:12.000233 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:12.052623 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:12.105698 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:21:12.113469 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:12.139145 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:12.146785 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:12.148990 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:12.163794 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:12.234016 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:12.324982 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:12.345541 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:12.363829 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:12.409267 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:12.415846 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:12.419713 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:12.434136 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:12.445230 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:12.449326 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:12.453538 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:12.455801 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:12.459525 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:12.461196 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:12.461800 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:12.470132 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:12.501209 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:12.589337 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:12.635993 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:12.662742 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:12.664981 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:12.683387 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:12.699247 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:12.720259 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:12.777078 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:12.813086 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:12.815557 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:12.837667 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:12.908334 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:12.927436 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:12.943694 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:12.958355 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:12.970749 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:12.981694 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:12.987067 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:13.034412 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:13.051236 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:13.126317 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 14:21:13.134953 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:13.151069 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:13.154896 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:13.156345 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:13.161460 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:13.168396 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:13.336540 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:13.350183 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:13.382151 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:13.386242 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:13.387054 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:13.431787 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:13.444145 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:13.458483 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:13.474794 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:13.481122 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:13.484242 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:13.489141 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:13.534430 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:21:13.567668 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:13.570816 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:13.598203 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:13.625171 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:13.654194 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:13.671042 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:13.675935 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:13.675935 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:13.701807 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:13.770175 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:13.770145 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:13.850824 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:13.863920 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:13.890652 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 14:21:13.955060 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:14.045884 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:14.056430 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:14.078869 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:14.093135 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:14.096560 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:14.107504 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:14.109995 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:14.134835 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:14.163655 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:14.261225 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:14.271209 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:14.292933 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:14.316422 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:14.319319 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:14.337086 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:14.345775 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:14.355571 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:14.376271 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:14.382567 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:14.388481 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:14.415505 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:14.449501 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:14.510622 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:14.556200 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:14.576189 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:14.576431 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:14.589148 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:14.607694 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:14.608428 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:14.616322 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:14.735033 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:14.788625 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:14.795655 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:14.820469 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:14.828367 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:14.841495 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:14.852885 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:14.853508 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:14.853754 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:21:14.865427 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:14.875531 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 14:21:14.900560 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:14.935964 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:14.940306 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:14.942895 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:14.958624 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:14.973263 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:14.991423 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:15.002020 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:15.010794 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:15.058306 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:15.087654 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:15.092897 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:15.096426 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:15.108958 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:15.311990 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:15.323348 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:15.333086 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:15.335345 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:15.354965 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:15.359548 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:15.388566 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:15.389897 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:15.418048 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:15.424474 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:15.451012 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:15.471217 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:15.488018 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:15.518229 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 14:21:15.587000 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:15.605626 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:15.606020 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:15.613512 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:15.615348 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:15.670720 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:15.772933 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:15.832732 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:15.832895 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:15.846304 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:15.849292 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:15.866506 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:15.884862 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:15.895787 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:15.914233 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:15.950114 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:15.968546 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:16.038465 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:16.040707 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:16.112262 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240503 14:21:16.210999 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:16.215231 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:16.220871 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:16.255896 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:16.265085 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 14:21:16.307477 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:16.317327 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:16.318812 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:16.321300 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:16.323334 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:16.328051 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:16.334806 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:16.349947 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:16.376005 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:16.392508 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:21:16.394987 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:16.473055 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:16.484524 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:16.514592 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:16.524497 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:16.549541 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:16.565053 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:16.580837 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:16.589162 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:16.593142 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:16.640329 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:16.669610 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:16.805747 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:16.808859 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:16.812774 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:16.813267 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:16.814386 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:16.818042 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:16.819118 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:16.853763 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:16.873651 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:16.881664 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:16.967397 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:16.988013 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:17.019953 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:17.034662 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:17.049367 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:17.050223 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:17.052500 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:17.058354 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:17.064391 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:17.064447 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:17.138562 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240503 14:21:17.217386 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:17.273841 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:17.276064 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:17.281075 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:17.281482 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:17.287369 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:17.302572 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:17.332607 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:17.336350 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:17.339903 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:17.375797 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:17.396848 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:17.429872 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:17.453243 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:17.469936 21328 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:17.470155 21328 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:17.470948 21328 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:17.471707 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:21:17.471974 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:21:17.472380 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:17.472868 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:17.472961 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:17.473198 21328 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:17.475934 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:17.498920 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:17.520375 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:17.529580 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:17.544111 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:17.563110 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:17.582231 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:17.588176 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:17.607338 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:17.652848 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:17.720072 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:17.762902 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:17.773669 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:17.792007 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:21:17.812705 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:17.819022 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:17.828130 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:17.828538 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:17.839591 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:17.855687 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:17.908205 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:17.959532 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:17.989694 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:18.045893 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:18.052335 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:18.059615 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:18.077252 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:18.090662 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:18.117745 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:18.147320 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:18.161317 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:18.215390 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:18.426981 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:18.446097 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:18.490211 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:18.491024 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:18.495081 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:18.502677 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:18.509646 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:18.519788 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:18.525794 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:18.527431 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:18.564957 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:18.578240 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:18.627358 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:18.631690 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:18.675107 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:21:18.708667 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:18.725131 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 14:21:18.729743 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:18.730089 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:18.748214 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:18.750519 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:18.750898 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:18.762866 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:18.794224 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:18.971650 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:18.978096 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:18.993048 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:19.006762 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 14:21:19.009114 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:19.009296 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:19.010627 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:19.014988 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:19.022095 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:19.029263 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:19.036723 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:19.037066 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:19.038151 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:19.074982 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:19.117548 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:19.135380 21638 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:19.135574 21638 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:19.136341 21638 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:19.137837 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:19.138037 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:19.138116 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:19.138347 21638 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:19.209965 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:19.214294 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:19.229038 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:19.229725 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:19.244200 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:19.252825 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:19.318028 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:19.402431 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:19.442446 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:19.464160 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:19.473343 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:19.474349 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:19.480994 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:19.493731 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:19.498441 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:19.536638 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:19.571635 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:19.578260 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:19.632577 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:19.679138 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:19.712769 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:19.721415 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:21:19.724392 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:19.728163 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:19.742837 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:19.743263 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:21:19.758711 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:19.759433 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:19.763008 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:19.791751 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:19.902025 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:19.958662 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:19.969219 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:19.971410 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 14:21:19.973294 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:19.989233 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:19.991880 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:20.006062 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:20.009668 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:20.028864 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:20.045490 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:20.067113 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:20.086122 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:20.104636 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:20.124635 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:20.144215 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:20.172286 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:21:20.192996 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:20.211467 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:20.224431 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:20.233736 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:20.239084 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:20.271211 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:20.333902 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:20.474043 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:20.587152 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:20.631078 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:20.662386 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:20.680548 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:20.681613 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:20.682399 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:20.682865 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:20.687283 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:20.700798 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:20.757018 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:20.823513 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:20.842864 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:21:20.894598 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:20.901387 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:20.903450 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:20.910619 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:20.920325 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:20.920436 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:20.939006 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:20.997754 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:21.066773 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:21.073611 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:21.087360 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:21.090605 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:21.121220 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:21.136700 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:21.137967 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:21.143777 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 125 similar messages]
I20240503 14:21:21.154592 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:21.156519 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:21.172606 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:21.204680 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:21.273695 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:21.292429 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 14:21:21.326761 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:21.358264 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:21.382769 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:21.393254 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:21.405763 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240503 14:21:21.442991 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:21.456513 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:21.456698 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:21.460211 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:21.500406 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:21.556557 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:21.559635 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:21.569993 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:21.604413 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:21.641968 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:21.675410 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 14:21:21.727250 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:21.729444 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:21.750593 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:21.760713 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:21.761284 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:21.764858 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:21.784842 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:21.798377 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:21.835753 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:21.838851 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:21.949035 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:21.958822 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:21.960093 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:21.976269 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:21.986197 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:21.995821 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:22.002441 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:22.007148 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:22.027765 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:22.070683 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:22.163044 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:21:22.187042 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:22.194936 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:22.217401 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:22.229008 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:22.230896 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:22.254009 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:22.266650 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:22.276010 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:22.287488 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:22.332194 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:22.337738 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:22.373119 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:22.414167 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:22.437391 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:22.460068 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:22.470455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:22.471724 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:22.475440 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:22.487394 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:22.489863 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:22.521816 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:22.532361 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:22.547994 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:22.566934 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:22.676200 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:22.732041 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:22.748898 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:22.824735 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:22.891441 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:22.891988 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:22.919247 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:22.931861 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:22.936990 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:22.937000 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:22.972224 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:23.002761 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:23.025333 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:23.080344 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:23.097744 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:23.107730 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:23.107769 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:23.128216 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:23.130179 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:23.130707 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:23.134757 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:23.195071 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:23.257709 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:23.350265 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:23.367415 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:23.369768 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:23.374606 23222 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:23.374773 23222 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:23.375512 23222 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:21:23.377195 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:23.381539 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:23.383466 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:23.383611 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:21:23.384151 23222 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:23.387984 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:23.401938 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:23.406841 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:23.485440 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:23.539206 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:23.587982 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:23.596817 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:23.601822 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:23.624260 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:23.632184 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:23.633868 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:23.640936 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:23.668087 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:23.674219 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:23.681279 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:23.830475 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:23.838070 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:23.840167 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:23.869268 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:23.878310 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:23.889806 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:23.893626 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:23.906306 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:23.975008 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:21:24.007807 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 14:21:24.031095 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 14:21:24.106423 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:24.113514 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:24.114223 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:24.123171 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:24.123904 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:24.135273 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:24.145038 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:24.155495 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:24.164954 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:24.230504 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:24.235483 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:24.236315 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:24.303577 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:24.314091 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:24.338380 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:21:24.339424 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:24.362151 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:24.363293 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:24.365414 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:24.375439 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:24.396726 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:24.402102 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:24.406069 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:24.463068 21328 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:24.463297 21328 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:24.464144 21328 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:21:24.466691 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:24.468765 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:24.468885 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
W20240503 14:21:24.469027 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:24.469410 21328 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:24.517952 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:24.519958 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:24.593859 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:24.601763 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:24.606781 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:24.623531 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:24.628657 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:24.632211 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:24.635236 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:24.636986 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:24.651836 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:24.684180 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:24.700055 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:24.739795 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:24.789564 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:24.815815 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:24.854656 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:24.957193 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:24.972412 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240503 14:21:24.994347 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:25.023056 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:25.043637 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:25.047809 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:25.057580 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:25.064746 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:25.098223 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:25.098907 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:25.173185 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:25.213873 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:25.231146 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:25.238562 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:25.255287 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:25.271370 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:25.278378 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:25.282634 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:25.289770 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:25.290303 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:25.293946 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:25.315151 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:25.358549 21328 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:25.358760 21328 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:25.359849 21328 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:25.362789 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:21:25.363013 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:21:25.366509 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:25.367436 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:25.367558 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:25.368178 21328 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:25.390589 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:25.502439 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 14:21:25.504890 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:25.522368 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:25.526882 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:25.546419 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:25.546830 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:25.583407 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:25.661476 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:25.710544 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:25.711133 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:25.732592 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:25.748219 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:25.755843 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:25.758848 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:25.761651 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:25.762483 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:25.804730 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:25.878324 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:25.941668 2497 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:25.986909 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:25.999845 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:26.006760 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:26.019771 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:26.026930 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:26.034399 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:26.067327 2367 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:26.138695 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:26.178431 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240503 14:21:26.179510 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:26.179652 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:26.187060 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:26.207778 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:26.209659 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:26.229586 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:26.233562 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:26.241433 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:26.274072 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:26.285498 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:26.300741 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:26.425024 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:26.461354 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:26.486783 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:26.487066 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:26.493422 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:26.498479 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:26.505973 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:26.550496 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:21:26.551064 2627 consensus_peers.cc:485] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:26.623950 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 14:21:26.689395 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:26.710705 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:26.722235 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:26.724479 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:26.731632 2627 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:26.743446 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:26.744081 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:26.746039 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:26.769654 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:26.772303 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:26.775991 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:26.777801 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:26.818738 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:26.884589 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:26.898433 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:26.915071 2236 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:27.067034 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:27.077136 2497 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:27.094468 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:27.104966 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:27.111227 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:27.133666 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:27.147547 2627 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:27.161731 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:27.168952 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:27.169965 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:27.183569 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:27.189708 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:27.218290 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:27.223743 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:27.227967 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:27.256587 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240503 14:21:27.272869 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:27.301054 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:27.315783 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:27.433158 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:27.441442 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:27.455279 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:27.455780 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:21:27.458061 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:27.460963 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:27.476994 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:27.511076 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:27.566998 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:27.637199 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:27.647848 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:27.664878 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:27.665560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:27.673550 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:27.692636 23464 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:27.692860 23464 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:27.693558 23464 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:27.697338 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:27.703150 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:27.707541 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:27.707696 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:27.708174 23464 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:27.719667 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:27.731423 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:27.755676 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:27.758237 2497 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:21:27.779827 23622 raft_consensus.cc:998] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.004s)
I20240503 14:21:27.780771 23622 consensus_queue.cc:237] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:27.792902 4333 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:27.793274 2691 raft_consensus.cc:1261] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:27.793560 23989 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:27.793902 23981 consensus_queue.cc:1035] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:27.797338 23981 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:27.798449 4333 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:27.799280 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:27.802820 2691 raft_consensus.cc:2932] T 5f631c7713d24f4caf7550bb05147615 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:27.804071 2154 catalog_manager.cc:5484] T 5f631c7713d24f4caf7550bb05147615 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:27.814800 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:27.824190 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:27.825584 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f631c7713d24f4caf7550bb05147615: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:27.858072 24016 raft_consensus.cc:998] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.406s)
I20240503 14:21:27.859382 24016 consensus_queue.cc:237] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
W20240503 14:21:27.860635 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:27.875041 4333 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:27.875782 21328 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:21:27.876745 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:27.880465 2430 raft_consensus.cc:1261] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:27.881057 24021 consensus_queue.cc:1035] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:27.883651 24016 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:27.884755 2430 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:27.892261 4333 raft_consensus.cc:2932] T af99824d1853471b867e4ff1f01e9367 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:27.894109 2157 catalog_manager.cc:5484] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:27.910533 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:27.920132 23545 raft_consensus.cc:998] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.167s)
I20240503 14:21:27.923008 23545 consensus_queue.cc:237] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:21:27.924579 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:27.928298 2299 raft_consensus.cc:1261] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:21:27.928815 2562 raft_consensus.cc:1261] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:21:27.929051 24028 consensus_queue.cc:1035] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:21:27.930203 24028 consensus_queue.cc:1035] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:21:27.969027 24033 raft_consensus.cc:998] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.229s)
I20240503 14:21:27.970446 24033 consensus_queue.cc:237] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:27.981148 24030 raft_consensus.cc:2932] T ff59573e2c8f472585b96ca0a0fd75b1 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:27.983827 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:27.981230 2562 raft_consensus.cc:2932] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:27.994736 2158 catalog_manager.cc:5484] T ff59573e2c8f472585b96ca0a0fd75b1 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:27.997171 2299 raft_consensus.cc:2932] T ff59573e2c8f472585b96ca0a0fd75b1 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:28.001365 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:28.002923 23464 raft_consensus.cc:998] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.375s)
I20240503 14:21:28.003866 23464 consensus_queue.cc:237] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
W20240503 14:21:28.013005 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:28.023941 2691 raft_consensus.cc:1261] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:28.024655 24036 consensus_queue.cc:1035] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:28.027560 24036 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:21:28.031340 2691 raft_consensus.cc:2932] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:21:28.036459 2154 catalog_manager.cc:5484] T 6325a400641b4d25b95592abb4af7d2d P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:21:28.041769 2430 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:28.042834 4333 raft_consensus.cc:1261] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:28.042881 24036 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:28.043412 24034 consensus_queue.cc:1035] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 14:21:28.050895 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:28.052021 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:28.054430 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 14:21:28.060631 24036 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:28.064779 4334 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:28.069396 2154 catalog_manager.cc:5484] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:28.071120 2430 raft_consensus.cc:2932] T aba6537cdb87400cb8b108b1fbfa6c0f P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:28.120829 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:28.137673 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:28.160636 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:28.228600 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:28.241935 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:28.242484 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:28.248390 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:28.264775 23545 raft_consensus.cc:998] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.454s)
I20240503 14:21:28.265444 23545 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
W20240503 14:21:28.267306 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:28.271337 24030 raft_consensus.cc:998] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.426s)
I20240503 14:21:28.272079 24030 consensus_queue.cc:237] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:28.274761 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:28.287029 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:28.288750 2430 raft_consensus.cc:1261] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:28.289508 24028 consensus_queue.cc:1035] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:21:28.296536 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:28.312603 24034 raft_consensus.cc:998] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.506s)
I20240503 14:21:28.313321 24034 consensus_queue.cc:237] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:21:28.317359 2431 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:21:28.318051 24030 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:21:28.328334 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:28.329661 2299 raft_consensus.cc:1261] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:28.330233 24034 consensus_queue.cc:1035] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:28.330569 24030 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:28.331862 2431 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:28.338152 2158 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:28.338640 23545 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:21:28.338869 2432 raft_consensus.cc:2932] T 832f07a6b12740d08fa67d653b3818bf P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:21:28.352556 2691 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:28.354593 2431 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:21:28.354753 2628 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763): Couldn't send request to peer 285488446ac445c5a5dfe654bfaf9835. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b63f87fb384bb2a48044bcad87c282. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:28.355777 24030 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:28.359156 24030 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:28.363754 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 61b63f87fb384bb2a48044bcad87c282 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:28.364038 2431 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:28.370908 2153 catalog_manager.cc:5484] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } }
I20240503 14:21:28.380167 2153 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 6 to 7, VOTER 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:28.392205 24036 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:28.394793 2299 raft_consensus.cc:2932] T 57c4a133b5c646f78b07f37dc85904f6 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:28.402300 2158 catalog_manager.cc:5484] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 6 to 7, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:21:28.464948 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:28.485025 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:28.541407 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:28.556136 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:28.566504 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:28.585614 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:28.612186 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:28.630322 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:28.724112 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:28.730718 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:28.733651 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:28.744863 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:28.758402 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:28.804838 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:28.811774 24168 ts_tablet_manager.cc:902] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
I20240503 14:21:28.825479 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:28.826087 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:28.830045 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:28.836169 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:28.863539 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:21:28.878952 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:28.949793 24168 tablet_copy_client.cc:308] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:21:28.950592 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 61b63f87fb384bb2a48044bcad87c282 from peer 285488446ac445c5a5dfe654bfaf9835 ({username='slave'} at 127.20.99.6:38575)
I20240503 14:21:28.950771 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 61b63f87fb384bb2a48044bcad87c282 from peer 285488446ac445c5a5dfe654bfaf9835 at {username='slave'} at 127.20.99.6:38575: session id = 285488446ac445c5a5dfe654bfaf9835-61b63f87fb384bb2a48044bcad87c282
I20240503 14:21:28.955124 2712 tablet_copy_source_session.cc:205] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:28.956833 24168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b63f87fb384bb2a48044bcad87c282. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:21:28.958791 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:28.964025 24168 tablet_copy_client.cc:783] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:28.978904 24168 tablet_copy_client.cc:647] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:28.980810 24168 tablet_copy_client.cc:523] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:28.984333 24168 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap starting.
W20240503 14:21:28.986411 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 14:21:28.989337 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 14:21:28.994132 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 832f07a6b12740d08fa67d653b3818bf: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:28.999876 24168 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:29.000252 24168 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: Bootstrap complete.
I20240503 14:21:29.000527 24168 ts_tablet_manager.cc:1367] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20240503 14:21:29.001181 24168 raft_consensus.cc:348] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:29.001358 24168 raft_consensus.cc:729] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285488446ac445c5a5dfe654bfaf9835, State: Initialized, Role: FOLLOWER
I20240503 14:21:29.001598 24168 consensus_queue.cc:260] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:29.002040 24168 ts_tablet_manager.cc:1398] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:21:29.003101 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session 285488446ac445c5a5dfe654bfaf9835-61b63f87fb384bb2a48044bcad87c282 received from {username='slave'} at 127.20.99.6:38575
I20240503 14:21:29.003242 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session 285488446ac445c5a5dfe654bfaf9835-61b63f87fb384bb2a48044bcad87c282 on tablet 61b63f87fb384bb2a48044bcad87c282 with peer 285488446ac445c5a5dfe654bfaf9835
W20240503 14:21:29.055343 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:29.066792 4381 maintenance_manager.cc:419] P 285488446ac445c5a5dfe654bfaf9835: Scheduling UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282): 4837 bytes on disk
I20240503 14:21:29.067399 4272 maintenance_manager.cc:643] P 285488446ac445c5a5dfe654bfaf9835: UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240503 14:21:29.131645 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:29.151945 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:29.158111 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:29.160450 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:29.178442 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:29.195730 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:21:29.241117 4334 raft_consensus.cc:1203] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240503 14:21:29.255220 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:29.294342 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:29.295389 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:29.313549 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:29.327811 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:29.377240 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:29.493073 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:29.503831 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:29.568732 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:29.576083 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:29.583649 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:29.615757 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:29.616058 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:29.728981 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:29.763449 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:29.786605 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:29.796770 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:29.803117 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:29.806392 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:29.809528 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:29.823331 24016 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:29.823516 24016 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:29.824328 24016 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:29.824671 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:21:29.824941 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:21:29.827504 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:29.827683 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:29.827769 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:29.828172 24016 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:29.837262 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:29.841148 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:29.853564 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:29.855973 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:29.881114 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:29.897974 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:29.918440 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:29.971550 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:30.009429 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:30.027127 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:30.033260 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:30.043536 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:30.046877 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:21:30.055698 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:30.062134 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:30.093259 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f631c7713d24f4caf7550bb05147615: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:30.208204 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:30.220705 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 14:21:30.251686 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:30.267670 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:30.268882 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:30.281344 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:30.293749 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:30.298334 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:30.323890 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:21:30.356010 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:30.373378 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:30.400861 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:30.471037 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:30.474872 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:21:30.484861 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:30.486186 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:30.486661 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:30.488416 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:30.553624 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:30.566434 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:30.590986 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:21:30.616330 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:21:30.649600 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:30.699972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:30.707228 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:30.724220 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:30.732267 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:30.739408 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:30.764483 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:30.783632 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:21:30.862783 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:30.880007 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:30.965236 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:30.993741 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:31.004009 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:31.006707 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:31.014019 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:31.017110 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 14:21:31.023459 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:31.097090 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:31.097525 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:31.164665 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:31.197572 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 125 similar messages]
I20240503 14:21:31.210918 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:31.218741 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:31.228857 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:31.244081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:31.244750 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:31.245615 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:31.259536 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:31.308871 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:31.328069 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 14:21:31.370915 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:31.372913 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:31.478945 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:31.508970 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:31.573608 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:31.578902 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 85 similar messages]
W20240503 14:21:31.579725 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:31.677151 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:31.678189 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:31.685102 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:31.696660 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:31.707046 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:31.738241 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:31.805716 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:31.843338 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:31.876092 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:31.877595 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:31.903249 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:31.916741 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:31.933533 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:31.933845 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:31.934965 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:31.951632 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:31.966169 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:32.024590 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:32.084957 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:32.114984 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:32.137452 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:32.137830 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:32.151693 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:32.154194 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:32.178746 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:32.182261 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:32.210587 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f631c7713d24f4caf7550bb05147615: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:32.304181 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:32.326139 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240503 14:21:32.360702 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:32.378190 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:21:32.378974 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:32.390764 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:32.392664 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:32.414463 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:32.414525 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:32.463454 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:32.475313 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:32.516237 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:32.577558 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:32.627868 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:32.633734 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:32.643425 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:32.649695 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:32.649811 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:32.671478 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:32.675566 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:32.693312 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:32.759439 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:21:32.835395 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:21:32.841423 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:32.866616 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:32.868175 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:32.868265 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:32.878460 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:32.896492 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:32.896955 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:32.957266 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:21:32.961763 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:33.050276 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:33.101418 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:33.104579 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:33.116235 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:33.125023 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:33.132345 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:33.136318 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:33.174476 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:21:33.199892 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:33.203570 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:33.206861 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:21:33.336946 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:33.348592 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:33.349557 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:33.356171 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:33.364180 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:33.369405 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:33.411459 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:33.493562 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:33.558899 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:33.601063 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:33.661034 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:33.684904 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 14:21:33.761806 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:33.768319 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 14:21:33.782233 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:33.794006 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:33.814751 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:33.816437 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:33.834199 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:33.856678 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:33.944813 2368 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:33.970453 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:33.980146 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:33.994045 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:34.001139 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:34.007555 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:34.014768 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:34.028606 4271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 14:21:34.029253 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:34.040154 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:34.043733 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:34.053717 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:34.077509 2367 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:34.218384 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:34.256974 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:34.270547 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:34.274024 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:34.301072 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:34.326239 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:34.341831 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 14:21:34.346624 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:34.406141 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:34.440855 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 14:21:34.475621 2368 consensus_peers.cc:485] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:34.510178 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:34.540735 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 14:21:34.560806 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:34.567934 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:34.578352 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:34.598261 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:34.600493 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:34.602454 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:34.613556 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:34.657316 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:34.776576 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:34.792311 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:34.792948 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:34.813792 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:34.827131 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:34.833734 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:34.910404 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:34.912197 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:34.998507 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:34.999893 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:35.014883 2498 consensus_peers.cc:462] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:35.029665 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:35.029798 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:35.030328 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:35.060521 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:35.066320 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:35.091478 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:21:35.118103 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:21:35.150050 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:35.185868 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:21:35.197672 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:35.221133 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:21:35.222585 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:35.245570 24846 raft_consensus.cc:998] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2: Attempting to remove follower 7146de39e348442da4a363e95fbe0789 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7146de39e348442da4a363e95fbe0789 for more than 300 seconds (300.517s)
I20240503 14:21:35.246641 24846 consensus_queue.cc:237] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:21:35.259642 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:35.261036 2299 raft_consensus.cc:1261] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:21:35.261745 25074 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:21:35.263734 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:35.297199 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:35.301800 2562 raft_consensus.cc:1261] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:21:35.304572 24846 consensus_queue.cc:1035] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:21:35.307826 2562 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:35.308595 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:35.308854 25119 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:35.312376 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:35.314285 2299 raft_consensus.cc:2932] T 470b425f5cef494ab3334fe2b85e1c4e P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:35.318712 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:35.320327 2158 catalog_manager.cc:5484] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 5 to 6, VOTER 7146de39e348442da4a363e95fbe0789 (127.20.99.5) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:21:35.365422 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 14:21:35.383335 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:35.403690 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:35.408648 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:35.428326 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:21:35.434756 24481 raft_consensus.cc:998] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.190s)
I20240503 14:21:35.435555 24481 consensus_queue.cc:237] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:35.437335 4333 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:21:35.437357 2299 raft_consensus.cc:1261] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:21:35.437935 25141 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:35.438155 25075 consensus_queue.cc:1035] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:35.440737 25141 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:35.441617 2299 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:35.444162 4333 raft_consensus.cc:2932] T a9a474791f9b4585aa0114162705c96f P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:35.446805 2158 catalog_manager.cc:5484] T a9a474791f9b4585aa0114162705c96f P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 7 to 8, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:35.514979 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:35.546644 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:35.566165 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:35.592800 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:35.617102 2237 consensus_peers.cc:462] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:35.617830 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:35.623239 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:35.637254 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:21:35.648962 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:35.650475 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:35.865551 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:35.943734 25172 raft_consensus.cc:998] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.071s)
I20240503 14:21:35.944480 25172 consensus_queue.cc:237] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
I20240503 14:21:35.947500 2691 raft_consensus.cc:1261] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:35.948175 25120 consensus_queue.cc:1035] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:21:35.948421 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:21:35.951387 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:21:35.951915 25184 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:35.952452 2691 raft_consensus.cc:2932] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:35.957461 2160 catalog_manager.cc:5484] T dc88c5dc94844f1c8400c31ad1165a32 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:21:35.960902 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:36.032395 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:36.032881 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:36.041257 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:36.058696 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:36.060669 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:36.065429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:36.097769 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:36.150656 25225 raft_consensus.cc:998] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.384s)
I20240503 14:21:36.151477 25225 consensus_queue.cc:237] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:36.153755 4333 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:36.154131 2691 raft_consensus.cc:1261] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Refusing update from remote peer 7f22b4d5b1b5466dbbdd344f0721c05f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:36.154471 24960 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:36.154999 24960 consensus_queue.cc:1035] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:36.157727 24960 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:36.161201 4333 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:36.166625 2152 catalog_manager.cc:5484] T 9ae190d0f4d843c990e6fdadcac677ed P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 1 leader_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:36.168941 2691 raft_consensus.cc:2932] T 9ae190d0f4d843c990e6fdadcac677ed P c37183c2965f46a28847c1262eb97829 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:36.203611 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 14:21:36.232630 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 14:21:36.249982 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:36.274149 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:36.277709 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:36.281016 25172 raft_consensus.cc:998] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.311s)
I20240503 14:21:36.281749 25172 consensus_queue.cc:237] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:36.293900 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:36.301527 4333 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:36.302223 25120 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:36.308321 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:36.314771 2561 raft_consensus.cc:1261] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:21:36.316301 25120 consensus_queue.cc:1035] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:21:36.317026 25184 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:36.318079 4333 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:36.324203 2160 catalog_manager.cc:5484] T 6c94404d8d4f45d59958ae6feb36fc24 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 1 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:36.356295 2562 raft_consensus.cc:2932] T 6c94404d8d4f45d59958ae6feb36fc24 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:36.388296 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6c94404d8d4f45d59958ae6feb36fc24: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:36.447705 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 14:21:36.461633 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:36.486181 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:36.513789 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 14:21:36.556892 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:36.578943 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:36.583961 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:36.595835 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:36.603127 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:36.632458 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 14:21:36.689198 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:36.766049 4271 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:36.798861 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:36.812455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:36.812698 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 14:21:36.813093 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 14:21:36.816817 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:36.835790 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:36.841092 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:36.968183 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:36.986029 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:36.991271 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 14:21:37.030673 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:37.033170 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:37.037003 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:37.037351 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:37.037511 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:37.122753 2498 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:37.137499 25141 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:37.137693 25141 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:37.138275 25141 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:37.139751 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:37.139863 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:37.139986 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:21:37.140197 25141 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:37.217078 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:37.227741 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:37.242385 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:37.246109 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:37.251447 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:37.255535 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:37.342489 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:21:37.381804 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:37.415925 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:37.444675 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 14:21:37.473498 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6c94404d8d4f45d59958ae6feb36fc24: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:37.493690 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:21:37.494225 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:37.504880 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:37.519518 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:37.538069 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:37.544600 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:37.567138 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:21:37.583458 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:21:37.657847 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:37.667899 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:37.691648 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:21:37.707509 4271 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:37.736129 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:37.736570 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:37.772154 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:37.777590 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:37.800630 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:37.801087 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:21:37.974527 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:37.974624 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:37.984323 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:37.987452 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:37.988703 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:37.993395 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:38.005082 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:38.045316 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:21:38.045871 25141 raft_consensus.cc:998] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.254s)
I20240503 14:21:38.046916 25141 consensus_queue.cc:237] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } }
W20240503 14:21:38.054601 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:38.055593 2299 raft_consensus.cc:1261] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 14:21:38.059947 25544 consensus_queue.cc:1035] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:21:38.078015 25542 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:38.078727 2299 raft_consensus.cc:2932] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
W20240503 14:21:38.083027 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:21:38.085408 2158 catalog_manager.cc:5484] T abdff0b719a342cfbde746751bd2a89d P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 3 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } }
I20240503 14:21:38.088268 24635 raft_consensus.cc:998] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.082s)
I20240503 14:21:38.089538 24635 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:38.088774 25546 raft_consensus.cc:998] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Attempting to remove follower c2cf5d2244244a2db5ebc7dc46fd430d from the Raft config. Reason: Leader has been unable to successfully communicate with peer c2cf5d2244244a2db5ebc7dc46fd430d for more than 300 seconds (300.147s)
I20240503 14:21:38.091424 25546 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:38.100560 2691 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 14:21:38.102130 25549 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:21:38.102421 2430 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 14:21:38.103428 24635 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:21:38.107120 2560 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 14:21:38.110571 25549 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:21:38.115434 2430 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.120429 2691 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.123966 25549 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.114114 25555 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.127207 2562 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.137856 2159 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 8 to 10, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.137856 2155 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 7 to 9, VOTER c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8) evicted. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:38.152989 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:38.166719 4333 consensus_queue.cc:237] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:21:38.169023 2562 raft_consensus.cc:1261] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
W20240503 14:21:38.169627 4268 consensus_peers.cc:485] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 -> Peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): Couldn't send request to peer c37183c2965f46a28847c1262eb97829. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:38.169943 25549 consensus_queue.cc:1035] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:21:38.172398 24635 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:38.173511 2562 raft_consensus.cc:2932] T fda4460865854505b5220d7ec3b34c57 P b35c386497d046de9be5007a31ab7d10 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:38.176456 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fda4460865854505b5220d7ec3b34c57 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:38.179492 2160 catalog_manager.cc:5484] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 10 to 11, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) added. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
W20240503 14:21:38.268824 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:21:38.321849 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:38.337090 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:38.360276 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:38.361608 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:21:38.362761 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:38.383675 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:38.403962 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:38.467279 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:21:38.497676 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet abdff0b719a342cfbde746751bd2a89d: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:38.545270 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:38.592101 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:38.611302 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:38.623833 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:38.632462 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:38.635092 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:38.679850 25642 ts_tablet_manager.cc:902] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Initiating tablet copy from peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
W20240503 14:21:38.702922 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:38.703738 25642 tablet_copy_client.cc:308] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:42763
I20240503 14:21:38.704715 4355 tablet_copy_service.cc:140] P 285488446ac445c5a5dfe654bfaf9835: Received BeginTabletCopySession request for tablet fda4460865854505b5220d7ec3b34c57 from peer c37183c2965f46a28847c1262eb97829 ({username='slave'} at 127.20.99.4:48997)
I20240503 14:21:38.704893 4355 tablet_copy_service.cc:161] P 285488446ac445c5a5dfe654bfaf9835: Beginning new tablet copy session on tablet fda4460865854505b5220d7ec3b34c57 from peer c37183c2965f46a28847c1262eb97829 at {username='slave'} at 127.20.99.4:48997: session id = c37183c2965f46a28847c1262eb97829-fda4460865854505b5220d7ec3b34c57
I20240503 14:21:38.708454 4355 tablet_copy_source_session.cc:205] T fda4460865854505b5220d7ec3b34c57 P 285488446ac445c5a5dfe654bfaf9835: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:38.710502 25642 ts_tablet_manager.cc:1875] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:21:38.716302 25642 ts_tablet_manager.cc:1888] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:21:38.716573 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4460865854505b5220d7ec3b34c57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:21:38.719210 25642 tablet_copy_client.cc:783] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:38.725348 25642 tablet_copy_client.cc:647] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:38.727097 25642 tablet_copy_client.cc:523] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:38.731988 25642 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
W20240503 14:21:38.737479 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 14:21:38.755035 25642 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:38.755523 25642 tablet_bootstrap.cc:492] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Bootstrap complete.
I20240503 14:21:38.755832 25642 ts_tablet_manager.cc:1367] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20240503 14:21:38.756640 25642 raft_consensus.cc:348] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:21:38.756950 25642 raft_consensus.cc:729] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:21:38.757238 25642 consensus_queue.cc:260] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:21:38.757792 25642 ts_tablet_manager.cc:1398] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:21:38.758507 4355 tablet_copy_service.cc:342] P 285488446ac445c5a5dfe654bfaf9835: Request end of tablet copy session c37183c2965f46a28847c1262eb97829-fda4460865854505b5220d7ec3b34c57 received from {username='slave'} at 127.20.99.4:48997
I20240503 14:21:38.758630 4355 tablet_copy_service.cc:433] P 285488446ac445c5a5dfe654bfaf9835: ending tablet copy session c37183c2965f46a28847c1262eb97829-fda4460865854505b5220d7ec3b34c57 on tablet fda4460865854505b5220d7ec3b34c57 with peer c37183c2965f46a28847c1262eb97829
I20240503 14:21:38.771232 4333 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:38.772781 2691 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 14:21:38.773279 25546 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:21:38.775236 25549 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.775828 2691 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.781038 2157 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.788530 4333 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:38.791878 2430 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 14:21:38.792258 2691 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 14:21:38.792501 25549 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:21:38.792984 25546 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:21:38.794585 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0eebf47c494dad8edf06197cdf15bc with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:37652
I20240503 14:21:38.794690 2430 raft_consensus.cc:858] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
W20240503 14:21:38.794785 2430 raft_consensus.cc:1565] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 NON_PARTICIPANT]: Could not prepare op 'term: 3 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:38.795296 2430 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 NON_PARTICIPANT]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:38.796234 25549 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:38.802369 2430 raft_consensus.cc:1467] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240503 14:21:38.802969 2430 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:38.805728 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fe0eebf47c494dad8edf06197cdf15bc with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:38.806795 2692 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:38.808197 2157 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 10 to 11, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
W20240503 14:21:38.813658 2145 catalog_manager.cc:4804] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): delete failed for tablet fe0eebf47c494dad8edf06197cdf15bc because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 9 but the committed config has opid_index of 10
I20240503 14:21:38.868407 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(fda4460865854505b5220d7ec3b34c57): 4583 bytes on disk
I20240503 14:21:38.869333 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(fda4460865854505b5220d7ec3b34c57) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 14:21:38.923377 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:38.927601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:38.946252 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:38.947752 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:38.950456 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:38.993834 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:39.027359 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 14:21:39.032146 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240503 14:21:39.054338 25227 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:39.054549 25227 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:39.055481 25227 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:21:39.056932 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:39.057663 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:39.057775 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
W20240503 14:21:39.057922 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:39.058039 25227 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:39.089066 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 14:21:39.101657 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:39.135174 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:39.137871 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:39.145021 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:39.145570 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:39.179343 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:39.197160 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 14:21:39.201508 2692 raft_consensus.cc:1203] T fda4460865854505b5220d7ec3b34c57 P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
W20240503 14:21:39.203045 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:39.316413 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 14:21:39.368613 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:39.380381 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:39.384418 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:39.387519 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:39.392341 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:39.406351 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:39.434429 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:39.630692 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:39.653188 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:39.757179 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:39.809376 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:39.821715 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:39.830009 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:39.830722 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:39.841832 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:39.849620 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:39.891691 2495 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.20.99.7:44387, user_credentials={real_user=slave}} blocked reactor thread for 49885.2us
W20240503 14:21:39.901902 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 14:21:39.964362 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:21:40.049022 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:40.057197 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:40.078393 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:40.078480 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:40.091461 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:40.102418 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:21:40.126960 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:21:40.156356 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:40.189594 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:21:40.198997 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:40.241207 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240503 14:21:40.242141 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:21:40.307025 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:40.323729 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:40.324296 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:40.343421 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:40.362093 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:40.374573 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:40.380512 2147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 14:21:40.426800 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:21:40.463380 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:40.563072 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:40.563797 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:40.595652 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:40.605765 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:40.606365 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:40.642088 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:21:40.656437 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:21:40.705420 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:40.723788 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:40.755295 4334 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:40.756999 2692 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 14:21:40.757515 25549 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:21:40.760073 25967 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:40.760700 2692 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:40.765486 2154 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 11 to 12, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:40.781095 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0eebf47c494dad8edf06197cdf15bc with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37652
I20240503 14:21:40.781853 4334 consensus_queue.cc:237] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:21:40.782125 25977 tablet_replica.cc:330] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:21:40.782335 25977 raft_consensus.cc:2229] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:21:40.782626 25977 raft_consensus.cc:2258] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:21:40.784411 2692 raft_consensus.cc:1261] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 14:21:40.784775 25977 ts_tablet_manager.cc:1875] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:21:40.784775 4269 consensus_peers.cc:485] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 -> Peer 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597): Couldn't send request to peer 7f22b4d5b1b5466dbbdd344f0721c05f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:40.785284 25967 consensus_queue.cc:1035] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:21:40.787642 25549 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:40.788677 2692 raft_consensus.cc:2932] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
W20240503 14:21:40.793313 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:40.794658 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet fe0eebf47c494dad8edf06197cdf15bc with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:40.794981 2154 catalog_manager.cc:5484] T fe0eebf47c494dad8edf06197cdf15bc P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 12 to 13, VOTER 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1) added. New cstate: current_term: 3 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:40.805440 25977 ts_tablet_manager.cc:1888] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 14:21:40.805562 25977 log.cc:1196] T fe0eebf47c494dad8edf06197cdf15bc P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe0eebf47c494dad8edf06197cdf15bc
I20240503 14:21:40.806301 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet fe0eebf47c494dad8edf06197cdf15bc (table rebalance_test_table_0 [id=92131433a8e24e6498f234341a021065]) successfully deleted
W20240503 14:21:40.854079 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:40.908509 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:40.916353 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:40.919272 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:40.930240 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:40.945492 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:41.035285 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:41.119669 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:41.129560 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:41.135756 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:41.136616 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:21:41.155217 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 14:21:41.156967 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:41.160439 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:41.184525 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:41.213619 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240503 14:21:41.261368 26062 ts_tablet_manager.cc:902] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Initiating tablet copy from peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
W20240503 14:21:41.265650 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:41.267789 25141 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:41.267974 25141 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:41.268589 25141 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:41.270687 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:41.275285 26062 tablet_copy_client.cc:308] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:42763
I20240503 14:21:41.276156 4355 tablet_copy_service.cc:140] P 285488446ac445c5a5dfe654bfaf9835: Received BeginTabletCopySession request for tablet fe0eebf47c494dad8edf06197cdf15bc from peer 7f22b4d5b1b5466dbbdd344f0721c05f ({username='slave'} at 127.20.99.1:33683)
I20240503 14:21:41.276320 4355 tablet_copy_service.cc:161] P 285488446ac445c5a5dfe654bfaf9835: Beginning new tablet copy session on tablet fe0eebf47c494dad8edf06197cdf15bc from peer 7f22b4d5b1b5466dbbdd344f0721c05f at {username='slave'} at 127.20.99.1:33683: session id = 7f22b4d5b1b5466dbbdd344f0721c05f-fe0eebf47c494dad8edf06197cdf15bc
W20240503 14:21:41.276451 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:41.276537 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:21:41.276741 25141 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:41.279704 4355 tablet_copy_source_session.cc:205] T fe0eebf47c494dad8edf06197cdf15bc P 285488446ac445c5a5dfe654bfaf9835: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:41.281111 26062 ts_tablet_manager.cc:1875] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:21:41.288161 26062 ts_tablet_manager.cc:1888] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:21:41.288426 26062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0eebf47c494dad8edf06197cdf15bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:21:41.292196 26062 tablet_copy_client.cc:783] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:41.299317 26062 tablet_copy_client.cc:647] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:41.301362 26062 tablet_copy_client.cc:523] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:41.304747 26062 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap starting.
I20240503 14:21:41.328002 26062 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:41.328539 26062 tablet_bootstrap.cc:492] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Bootstrap complete.
I20240503 14:21:41.328936 26062 ts_tablet_manager.cc:1367] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.001s
I20240503 14:21:41.329772 26062 raft_consensus.cc:348] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:21:41.330090 26062 raft_consensus.cc:729] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f22b4d5b1b5466dbbdd344f0721c05f, State: Initialized, Role: FOLLOWER
I20240503 14:21:41.330399 26062 consensus_queue.cc:260] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:21:41.333276 26062 ts_tablet_manager.cc:1398] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:21:41.334050 4355 tablet_copy_service.cc:342] P 285488446ac445c5a5dfe654bfaf9835: Request end of tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-fe0eebf47c494dad8edf06197cdf15bc received from {username='slave'} at 127.20.99.1:33683
I20240503 14:21:41.334214 4355 tablet_copy_service.cc:433] P 285488446ac445c5a5dfe654bfaf9835: ending tablet copy session 7f22b4d5b1b5466dbbdd344f0721c05f-fe0eebf47c494dad8edf06197cdf15bc on tablet fe0eebf47c494dad8edf06197cdf15bc with peer 7f22b4d5b1b5466dbbdd344f0721c05f
W20240503 14:21:41.360205 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 14:21:41.370617 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:41.383416 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:41.388199 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:41.392743 2347 maintenance_manager.cc:419] P 7f22b4d5b1b5466dbbdd344f0721c05f: Scheduling UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc): 4838 bytes on disk
I20240503 14:21:41.393404 2238 maintenance_manager.cc:643] P 7f22b4d5b1b5466dbbdd344f0721c05f: UndoDeltaBlockGCOp(fe0eebf47c494dad8edf06197cdf15bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 14:21:41.402318 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:41.425969 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:41.520368 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 14:21:41.524499 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:41.686318 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:41.706447 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 14:21:41.720778 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:41.726365 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 14:21:41.781399 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:41.784539 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240503 14:21:41.856326 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:41.860381 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:41.871788 2299 raft_consensus.cc:1203] T fe0eebf47c494dad8edf06197cdf15bc P 7f22b4d5b1b5466dbbdd344f0721c05f [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
I20240503 14:21:41.882932 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:41.890538 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:41.897092 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:42.001376 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:42.103864 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:42.113018 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:42.114058 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:42.130944 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:42.157788 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:42.163528 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:42.249109 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:42.329216 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:42.332535 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:42.337896 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:42.344048 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:42.351353 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:42.375327 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:21:42.404181 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:42.410301 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:42.438519 2368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 44112.3us
W20240503 14:21:42.452582 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fda4460865854505b5220d7ec3b34c57: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:42.580536 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:42.586480 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:42.594161 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:42.602870 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:42.612367 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 14:21:42.620257 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:21:42.626550 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:42.733716 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:21:42.735253 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:42.798641 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:42.828344 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:42.830657 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:21:42.841634 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:42.848815 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:21:42.851461 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:42.852423 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:42.871060 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:42.878672 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:42.970642 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:21:42.999212 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:21:43.061785 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:43.067970 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:43.082203 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:43.085319 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:43.123994 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:43.151854 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:43.175223 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:21:43.178315 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:21:43.190160 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:43.204645 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:43.266443 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 14:21:43.335460 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:43.337460 25141 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:43.337621 25141 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:43.338276 25141 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:43.340080 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 14:21:43.340569 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:43.342330 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:43.363440 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:43.371471 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:43.403085 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:43.404011 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:43.404312 25141 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:43.417385 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:43.426154 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:43.487818 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:21:43.587880 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:43.606078 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:43.611092 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:43.619680 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:43.636860 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:43.637033 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:43.804054 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:43.833063 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:43.840692 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:43.853549 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:43.867372 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:43.872220 26107 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:43.872407 26107 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:43.873461 26107 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:43.879400 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:21:43.879657 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:21:43.880735 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:43.881862 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:43.881978 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:43.882436 26107 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:43.890439 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:43.897708 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:43.935912 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:43.998642 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 14:21:44.067919 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:44.091686 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:44.096662 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:44.117053 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:44.118644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:44.122567 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:44.129504 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:44.220395 4268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 14:21:44.221925 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240503 14:21:44.236505 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 14:21:44.240310 2692 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:44.242254 4333 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:21:44.242789 26107 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:21:44.245422 26516 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:44.245998 4333 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:44.251521 2153 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 reported cstate change: config changed from index 7 to 8, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) evicted. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:44.264819 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b63f87fb384bb2a48044bcad87c282 with delete_type TABLET_DATA_TOMBSTONED (TS 685c6181561b4056abdecfd6d8bdb3b2 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:37652
I20240503 14:21:44.265507 26526 tablet_replica.cc:330] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: stopping tablet replica
I20240503 14:21:44.265790 26526 raft_consensus.cc:2229] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:21:44.266072 26526 raft_consensus.cc:2258] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:21:44.266436 2692 consensus_queue.cc:237] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:21:44.266770 26526 ts_tablet_manager.cc:1875] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:21:44.268487 4333 raft_consensus.cc:1261] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Refusing update from remote peer c37183c2965f46a28847c1262eb97829: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 14:21:44.268795 2625 consensus_peers.cc:485] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 -> Peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269): Couldn't send request to peer b35c386497d046de9be5007a31ab7d10. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:44.269156 26516 consensus_queue.cc:1035] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:21:44.271927 26516 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:21:44.276098 4333 raft_consensus.cc:2932] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:21:44.287201 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 61b63f87fb384bb2a48044bcad87c282 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:44.287402 2153 catalog_manager.cc:5484] T 61b63f87fb384bb2a48044bcad87c282 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 8 to 9, VOTER b35c386497d046de9be5007a31ab7d10 (127.20.99.3) added. New cstate: current_term: 1 leader_uuid: "c37183c2965f46a28847c1262eb97829" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } }
I20240503 14:21:44.297904 26526 ts_tablet_manager.cc:1888] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 14:21:44.298111 26526 log.cc:1196] T 61b63f87fb384bb2a48044bcad87c282 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting WAL directory at /tmp/dist-test-taskHTWOmZ/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714745515815701-20876-0/raft_consensus-itest-cluster/ts-1/wal/wals/61b63f87fb384bb2a48044bcad87c282
I20240503 14:21:44.300516 2145 catalog_manager.cc:4830] TS 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): tablet 61b63f87fb384bb2a48044bcad87c282 (table rebalance_test_table_3 [id=ee73b380a5364116800e5a1a41243d03]) successfully deleted
W20240503 14:21:44.350620 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 14:21:44.401436 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:44.416385 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:44.426924 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:44.430641 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:44.469257 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:44.488847 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:44.527074 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:44.651721 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:44.654505 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:44.672014 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:44.673669 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:44.694247 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:44.766960 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:44.778260 26594 ts_tablet_manager.cc:902] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Initiating tablet copy from peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091)
W20240503 14:21:44.780939 4268 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 14:21:44.793324 26594 tablet_copy_client.cc:308] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.4:39091
I20240503 14:21:44.794580 2712 tablet_copy_service.cc:140] P c37183c2965f46a28847c1262eb97829: Received BeginTabletCopySession request for tablet 61b63f87fb384bb2a48044bcad87c282 from peer b35c386497d046de9be5007a31ab7d10 ({username='slave'} at 127.20.99.3:37243)
I20240503 14:21:44.794785 2712 tablet_copy_service.cc:161] P c37183c2965f46a28847c1262eb97829: Beginning new tablet copy session on tablet 61b63f87fb384bb2a48044bcad87c282 from peer b35c386497d046de9be5007a31ab7d10 at {username='slave'} at 127.20.99.3:37243: session id = b35c386497d046de9be5007a31ab7d10-61b63f87fb384bb2a48044bcad87c282
I20240503 14:21:44.797657 2712 tablet_copy_source_session.cc:205] T 61b63f87fb384bb2a48044bcad87c282 P c37183c2965f46a28847c1262eb97829: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:44.799787 26594 ts_tablet_manager.cc:1875] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240503 14:21:44.803454 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:44.809535 26594 ts_tablet_manager.cc:1888] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:21:44.809834 26594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b63f87fb384bb2a48044bcad87c282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:21:44.813481 26594 tablet_copy_client.cc:783] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:44.820123 26594 tablet_copy_client.cc:647] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:44.822166 26594 tablet_copy_client.cc:523] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:44.826314 26594 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Bootstrap starting.
I20240503 14:21:44.846930 26594 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:44.847381 26594 tablet_bootstrap.cc:492] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Bootstrap complete.
I20240503 14:21:44.847663 26594 ts_tablet_manager.cc:1367] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Time spent bootstrapping tablet: real 0.021s user 0.023s sys 0.000s
I20240503 14:21:44.848266 26594 raft_consensus.cc:348] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:21:44.848460 26594 raft_consensus.cc:729] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35c386497d046de9be5007a31ab7d10, State: Initialized, Role: FOLLOWER
I20240503 14:21:44.848685 26594 consensus_queue.cc:260] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } }
I20240503 14:21:44.849166 26594 ts_tablet_manager.cc:1398] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:21:44.850104 2712 tablet_copy_service.cc:342] P c37183c2965f46a28847c1262eb97829: Request end of tablet copy session b35c386497d046de9be5007a31ab7d10-61b63f87fb384bb2a48044bcad87c282 received from {username='slave'} at 127.20.99.3:37243
I20240503 14:21:44.850248 2712 tablet_copy_service.cc:433] P c37183c2965f46a28847c1262eb97829: ending tablet copy session b35c386497d046de9be5007a31ab7d10-61b63f87fb384bb2a48044bcad87c282 on tablet 61b63f87fb384bb2a48044bcad87c282 with peer b35c386497d046de9be5007a31ab7d10
W20240503 14:21:44.903170 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:44.965566 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:44.978799 2608 maintenance_manager.cc:419] P b35c386497d046de9be5007a31ab7d10: Scheduling UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282): 4837 bytes on disk
I20240503 14:21:44.979596 2499 maintenance_manager.cc:643] P b35c386497d046de9be5007a31ab7d10: UndoDeltaBlockGCOp(61b63f87fb384bb2a48044bcad87c282) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":146,"lbm_reads_lt_1ms":4}
W20240503 14:21:44.980823 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 14:21:44.986377 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:44.987138 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:44.992944 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:45.012923 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:21:45.018122 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:45.020650 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 832f07a6b12740d08fa67d653b3818bf: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:45.055349 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:21:45.152654 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:45.196353 2560 raft_consensus.cc:1203] T 61b63f87fb384bb2a48044bcad87c282 P b35c386497d046de9be5007a31ab7d10 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240503 14:21:45.306116 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:45.321029 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:21:45.370354 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:21:45.371567 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:45.401103 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:45.405470 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:45.424717 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:45.428855 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:45.465005 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:45.466323 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:45.468619 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:45.472733 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:21:45.565850 4268 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 14:21:45.577903 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:45.633829 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:45.644533 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:45.661253 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:45.662748 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:45.666599 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 14:21:45.668339 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:45.679646 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:45.717509 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:45.865514 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:45.868264 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:45.897289 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:45.900715 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:45.902318 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:45.906808 2495 consensus_peers.cc:462] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:45.911880 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:45.952207 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:45.990898 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:46.116269 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:46.127866 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:46.133153 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:46.146932 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:46.149777 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:46.262957 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 14:21:46.304576 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 14:21:46.305737 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:46.339118 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:21:46.341078 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:46.362150 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:46.368167 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:46.371739 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:46.374608 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:46.430775 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:21:46.452194 2149 maintenance_manager.cc:419] P 13adeda3db494b478178ddd5ee8fdb98: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7339 bytes on disk
I20240503 14:21:46.457576 2148 maintenance_manager.cc:643] P 13adeda3db494b478178ddd5ee8fdb98: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":277,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":124,"thread_start_us":873,"threads_started":2,"wal-append.queue_time_us":331}
W20240503 14:21:46.473201 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:46.482259 25141 raft_consensus.cc:998] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: Attempting to remove follower d4cbc83202c8473d8c897582d30e5989 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d4cbc83202c8473d8c897582d30e5989 for more than 300 seconds (300.143s)
I20240503 14:21:46.482936 25141 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:46.484833 4333 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:21:46.485345 26850 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:21:46.487809 25141 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:46.488415 4333 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:46.493219 2158 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 reported cstate change: config changed from index 8 to 9, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
W20240503 14:21:46.496279 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 14:21:46.499109 2144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 14:21:46.501915 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:46.505072 2560 consensus_queue.cc:237] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
W20240503 14:21:46.507236 2496 consensus_peers.cc:485] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 -> Peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): Couldn't send request to peer 685c6181561b4056abdecfd6d8bdb3b2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:46.507279 4333 raft_consensus.cc:1261] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Refusing update from remote peer b35c386497d046de9be5007a31ab7d10: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:21:46.507752 25141 consensus_queue.cc:1035] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:21:46.509773 26850 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:46.510303 4333 raft_consensus.cc:2932] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:46.515278 2158 catalog_manager.cc:5484] T eb1d30b76c4d43d0a54a49892d09a6f8 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New cstate: current_term: 2 leader_uuid: "b35c386497d046de9be5007a31ab7d10" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:46.515705 2144 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet eb1d30b76c4d43d0a54a49892d09a6f8 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240503 14:21:46.517040 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:46.592187 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:46.610059 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:46.610617 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:46.615599 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:46.629881 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:46.630120 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:21:46.652495 26770 raft_consensus.cc:998] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Attempting to remove follower d4cbc83202c8473d8c897582d30e5989 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d4cbc83202c8473d8c897582d30e5989 for more than 300 seconds (300.011s)
I20240503 14:21:46.653246 26770 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } }
I20240503 14:21:46.662459 2300 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:21:46.663048 26770 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:21:46.681813 26770 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:46.683588 2300 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:46.688658 2160 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 8 to 9, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } }
I20240503 14:21:46.694192 4333 consensus_queue.cc:237] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
W20240503 14:21:46.696496 4269 consensus_peers.cc:485] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 -> Peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): Couldn't send request to peer 685c6181561b4056abdecfd6d8bdb3b2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:46.696732 2300 raft_consensus.cc:1261] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:21:46.697731 26770 consensus_queue.cc:1035] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:21:46.704978 26889 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:46.707989 2300 raft_consensus.cc:2932] T b359fb8d9afc4c9c856897a722100304 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:46.710750 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet b359fb8d9afc4c9c856897a722100304 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:46.713788 2160 catalog_manager.cc:5484] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 9 to 10, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
W20240503 14:21:46.807662 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 14:21:46.830081 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:46.843065 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:46.843173 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:46.844182 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:46.866376 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:46.918843 26675 raft_consensus.cc:998] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Attempting to remove follower d4cbc83202c8473d8c897582d30e5989 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d4cbc83202c8473d8c897582d30e5989 for more than 300 seconds (300.125s)
I20240503 14:21:46.920114 26675 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:21:46.925806 2300 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:21:46.926506 26675 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:21:46.929221 26932 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:46.929849 2300 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
W20240503 14:21:46.933465 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:46.935436 2152 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 10 to 11, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:46.940304 2431 consensus_queue.cc:237] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
W20240503 14:21:46.942309 2365 consensus_peers.cc:485] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091): Couldn't send request to peer c37183c2965f46a28847c1262eb97829. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:46.942481 2300 raft_consensus.cc:1261] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 685c6181561b4056abdecfd6d8bdb3b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:21:46.942934 26932 consensus_queue.cc:1035] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:21:46.944900 26675 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:46.948263 2145 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8bf4675882e64f35bc3dc9ee06ae366d with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:46.949038 2300 raft_consensus.cc:2932] T 8bf4675882e64f35bc3dc9ee06ae366d P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:46.950134 2152 catalog_manager.cc:5484] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2 reported cstate change: config changed from index 11 to 12, VOTER c37183c2965f46a28847c1262eb97829 (127.20.99.4) added. New cstate: current_term: 2 leader_uuid: "685c6181561b4056abdecfd6d8bdb3b2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } }
I20240503 14:21:47.013923 26955 ts_tablet_manager.cc:902] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Initiating tablet copy from peer b35c386497d046de9be5007a31ab7d10 (127.20.99.3:45269)
I20240503 14:21:47.016589 26955 tablet_copy_client.cc:308] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.3:45269
I20240503 14:21:47.017382 2582 tablet_copy_service.cc:140] P b35c386497d046de9be5007a31ab7d10: Received BeginTabletCopySession request for tablet eb1d30b76c4d43d0a54a49892d09a6f8 from peer 685c6181561b4056abdecfd6d8bdb3b2 ({username='slave'} at 127.20.99.2:53035)
I20240503 14:21:47.017593 2582 tablet_copy_service.cc:161] P b35c386497d046de9be5007a31ab7d10: Beginning new tablet copy session on tablet eb1d30b76c4d43d0a54a49892d09a6f8 from peer 685c6181561b4056abdecfd6d8bdb3b2 at {username='slave'} at 127.20.99.2:53035: session id = 685c6181561b4056abdecfd6d8bdb3b2-eb1d30b76c4d43d0a54a49892d09a6f8
I20240503 14:21:47.021438 2582 tablet_copy_source_session.cc:205] T eb1d30b76c4d43d0a54a49892d09a6f8 P b35c386497d046de9be5007a31ab7d10: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:47.023316 26955 ts_tablet_manager.cc:1875] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:21:47.035598 26955 ts_tablet_manager.cc:1888] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 14:21:47.036087 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1d30b76c4d43d0a54a49892d09a6f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:21:47.038031 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:47.040679 26955 tablet_copy_client.cc:783] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:47.049567 26955 tablet_copy_client.cc:647] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:47.051831 26955 tablet_copy_client.cc:523] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:47.057220 26955 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:21:47.061421 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:47.079996 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:47.088501 26955 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:47.089107 26955 tablet_bootstrap.cc:492] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap complete.
I20240503 14:21:47.089517 26955 ts_tablet_manager.cc:1367] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.001s
I20240503 14:21:47.090438 26955 raft_consensus.cc:348] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:47.090787 26955 raft_consensus.cc:729] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:21:47.091115 26955 consensus_queue.cc:260] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:47.091919 26955 ts_tablet_manager.cc:1398] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:21:47.092743 2582 tablet_copy_service.cc:342] P b35c386497d046de9be5007a31ab7d10: Request end of tablet copy session 685c6181561b4056abdecfd6d8bdb3b2-eb1d30b76c4d43d0a54a49892d09a6f8 received from {username='slave'} at 127.20.99.2:53035
I20240503 14:21:47.092857 2582 tablet_copy_service.cc:433] P b35c386497d046de9be5007a31ab7d10: ending tablet copy session 685c6181561b4056abdecfd6d8bdb3b2-eb1d30b76c4d43d0a54a49892d09a6f8 on tablet eb1d30b76c4d43d0a54a49892d09a6f8 with peer 685c6181561b4056abdecfd6d8bdb3b2
I20240503 14:21:47.095101 26770 raft_consensus.cc:998] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Attempting to remove follower d4cbc83202c8473d8c897582d30e5989 from the Raft config. Reason: Leader has been unable to successfully communicate with peer d4cbc83202c8473d8c897582d30e5989 for more than 300 seconds (300.314s)
I20240503 14:21:47.096037 26770 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } }
I20240503 14:21:47.104373 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:47.115589 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:47.120472 2300 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:21:47.124406 26889 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:21:47.132166 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:47.146564 26889 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:47.150769 2299 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:47.152463 2158 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 reported cstate change: config changed from index 10 to 11, VOTER d4cbc83202c8473d8c897582d30e5989 (127.20.99.7) evicted. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } }
I20240503 14:21:47.157948 4333 consensus_queue.cc:237] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
W20240503 14:21:47.160439 4269 consensus_peers.cc:485] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 -> Peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457): Couldn't send request to peer 685c6181561b4056abdecfd6d8bdb3b2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:21:47.160634 2299 raft_consensus.cc:1261] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Refusing update from remote peer 285488446ac445c5a5dfe654bfaf9835: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 14:21:47.161120 26889 consensus_queue.cc:1035] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:21:47.162971 26770 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:47.166055 2147 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 32927a23842b4725b9dfafbac47fe754 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 14:21:47.166484 2299 raft_consensus.cc:2932] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:47.172999 2160 catalog_manager.cc:5484] T 32927a23842b4725b9dfafbac47fe754 P 7f22b4d5b1b5466dbbdd344f0721c05f reported cstate change: config changed from index 11 to 12, VOTER 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2) added. New cstate: current_term: 2 leader_uuid: "285488446ac445c5a5dfe654bfaf9835" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } }
I20240503 14:21:47.258631 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(eb1d30b76c4d43d0a54a49892d09a6f8): 4923 bytes on disk
I20240503 14:21:47.259358 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(eb1d30b76c4d43d0a54a49892d09a6f8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
W20240503 14:21:47.272068 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 14:21:47.311784 26955 ts_tablet_manager.cc:902] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Initiating tablet copy from peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:21:47.313078 26955 tablet_copy_client.cc:308] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:42763
I20240503 14:21:47.313774 4355 tablet_copy_service.cc:140] P 285488446ac445c5a5dfe654bfaf9835: Received BeginTabletCopySession request for tablet b359fb8d9afc4c9c856897a722100304 from peer 685c6181561b4056abdecfd6d8bdb3b2 ({username='slave'} at 127.20.99.2:37851)
I20240503 14:21:47.313953 4355 tablet_copy_service.cc:161] P 285488446ac445c5a5dfe654bfaf9835: Beginning new tablet copy session on tablet b359fb8d9afc4c9c856897a722100304 from peer 685c6181561b4056abdecfd6d8bdb3b2 at {username='slave'} at 127.20.99.2:37851: session id = 685c6181561b4056abdecfd6d8bdb3b2-b359fb8d9afc4c9c856897a722100304
I20240503 14:21:47.317225 4355 tablet_copy_source_session.cc:205] T b359fb8d9afc4c9c856897a722100304 P 285488446ac445c5a5dfe654bfaf9835: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:47.318250 26955 ts_tablet_manager.cc:1875] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:21:47.323858 26955 ts_tablet_manager.cc:1888] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:21:47.324147 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b359fb8d9afc4c9c856897a722100304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:21:47.327280 26955 tablet_copy_client.cc:783] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:47.333663 26955 tablet_copy_client.cc:647] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:47.335605 26955 tablet_copy_client.cc:523] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:47.340973 26955 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:21:47.364403 26955 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:47.365187 26955 tablet_bootstrap.cc:492] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap complete.
I20240503 14:21:47.365554 26955 ts_tablet_manager.cc:1367] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240503 14:21:47.366616 26955 raft_consensus.cc:348] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:47.366928 26955 raft_consensus.cc:729] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:21:47.367259 26955 consensus_queue.cc:260] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:47.367847 26955 ts_tablet_manager.cc:1398] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:21:47.368630 4355 tablet_copy_service.cc:342] P 285488446ac445c5a5dfe654bfaf9835: Request end of tablet copy session 685c6181561b4056abdecfd6d8bdb3b2-b359fb8d9afc4c9c856897a722100304 received from {username='slave'} at 127.20.99.2:37851
I20240503 14:21:47.368753 4355 tablet_copy_service.cc:433] P 285488446ac445c5a5dfe654bfaf9835: ending tablet copy session 685c6181561b4056abdecfd6d8bdb3b2-b359fb8d9afc4c9c856897a722100304 on tablet b359fb8d9afc4c9c856897a722100304 with peer 685c6181561b4056abdecfd6d8bdb3b2
W20240503 14:21:47.389991 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:21:47.392525 27010 ts_tablet_manager.cc:902] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Initiating tablet copy from peer 685c6181561b4056abdecfd6d8bdb3b2 (127.20.99.2:43457)
W20240503 14:21:47.405751 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:47.420627 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:47.437623 27010 tablet_copy_client.cc:308] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.2:43457
I20240503 14:21:47.438829 2452 tablet_copy_service.cc:140] P 685c6181561b4056abdecfd6d8bdb3b2: Received BeginTabletCopySession request for tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer c37183c2965f46a28847c1262eb97829 ({username='slave'} at 127.20.99.4:45821)
I20240503 14:21:47.439079 2452 tablet_copy_service.cc:161] P 685c6181561b4056abdecfd6d8bdb3b2: Beginning new tablet copy session on tablet 8bf4675882e64f35bc3dc9ee06ae366d from peer c37183c2965f46a28847c1262eb97829 at {username='slave'} at 127.20.99.4:45821: session id = c37183c2965f46a28847c1262eb97829-8bf4675882e64f35bc3dc9ee06ae366d
I20240503 14:21:47.443470 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:47.444526 2452 tablet_copy_source_session.cc:205] T 8bf4675882e64f35bc3dc9ee06ae366d P 685c6181561b4056abdecfd6d8bdb3b2: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:47.449314 27010 ts_tablet_manager.cc:1875] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:21:47.457350 27010 ts_tablet_manager.cc:1888] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 14:21:47.457660 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4675882e64f35bc3dc9ee06ae366d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:21:47.461014 27010 tablet_copy_client.cc:783] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:47.475802 27010 tablet_copy_client.cc:647] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:47.478394 27010 tablet_copy_client.cc:523] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:21:47.488035 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:47.488955 27010 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Bootstrap starting.
I20240503 14:21:47.489629 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:47.499044 2431 raft_consensus.cc:1203] T eb1d30b76c4d43d0a54a49892d09a6f8 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240503 14:21:47.511767 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(b359fb8d9afc4c9c856897a722100304): 4838 bytes on disk
W20240503 14:21:47.512328 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 32927a23842b4725b9dfafbac47fe754: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:47.512485 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(b359fb8d9afc4c9c856897a722100304) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240503 14:21:47.514147 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:47.519778 27010 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:47.520304 27010 tablet_bootstrap.cc:492] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Bootstrap complete.
I20240503 14:21:47.521375 27010 ts_tablet_manager.cc:1367] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20240503 14:21:47.522274 27010 raft_consensus.cc:348] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:21:47.522559 27010 raft_consensus.cc:729] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37183c2965f46a28847c1262eb97829, State: Initialized, Role: FOLLOWER
I20240503 14:21:47.522935 27010 consensus_queue.cc:260] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } }
I20240503 14:21:47.523773 27010 ts_tablet_manager.cc:1398] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:21:47.529137 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:47.529734 2452 tablet_copy_service.cc:342] P 685c6181561b4056abdecfd6d8bdb3b2: Request end of tablet copy session c37183c2965f46a28847c1262eb97829-8bf4675882e64f35bc3dc9ee06ae366d received from {username='slave'} at 127.20.99.4:45821
I20240503 14:21:47.529841 2452 tablet_copy_service.cc:433] P 685c6181561b4056abdecfd6d8bdb3b2: ending tablet copy session c37183c2965f46a28847c1262eb97829-8bf4675882e64f35bc3dc9ee06ae366d on tablet 8bf4675882e64f35bc3dc9ee06ae366d with peer c37183c2965f46a28847c1262eb97829
I20240503 14:21:47.617009 26955 ts_tablet_manager.cc:902] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Initiating tablet copy from peer 285488446ac445c5a5dfe654bfaf9835 (127.20.99.6:42763)
I20240503 14:21:47.618152 26955 tablet_copy_client.cc:308] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.99.6:42763
I20240503 14:21:47.618809 4355 tablet_copy_service.cc:140] P 285488446ac445c5a5dfe654bfaf9835: Received BeginTabletCopySession request for tablet 32927a23842b4725b9dfafbac47fe754 from peer 685c6181561b4056abdecfd6d8bdb3b2 ({username='slave'} at 127.20.99.2:37851)
I20240503 14:21:47.618989 4355 tablet_copy_service.cc:161] P 285488446ac445c5a5dfe654bfaf9835: Beginning new tablet copy session on tablet 32927a23842b4725b9dfafbac47fe754 from peer 685c6181561b4056abdecfd6d8bdb3b2 at {username='slave'} at 127.20.99.2:37851: session id = 685c6181561b4056abdecfd6d8bdb3b2-32927a23842b4725b9dfafbac47fe754
I20240503 14:21:47.621613 4355 tablet_copy_source_session.cc:205] T 32927a23842b4725b9dfafbac47fe754 P 285488446ac445c5a5dfe654bfaf9835: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:21:47.622591 26955 ts_tablet_manager.cc:1875] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:21:47.629817 26955 ts_tablet_manager.cc:1888] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
W20240503 14:21:47.629812 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:21:47.630079 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32927a23842b4725b9dfafbac47fe754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:21:47.633198 26955 tablet_copy_client.cc:783] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Starting download of 5 data blocks...
I20240503 14:21:47.639672 26955 tablet_copy_client.cc:647] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Starting download of 1 WAL segments...
I20240503 14:21:47.642038 26955 tablet_copy_client.cc:523] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:21:47.645349 26955 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap starting.
I20240503 14:21:47.661232 2738 maintenance_manager.cc:419] P c37183c2965f46a28847c1262eb97829: Scheduling UndoDeltaBlockGCOp(8bf4675882e64f35bc3dc9ee06ae366d): 4923 bytes on disk
I20240503 14:21:47.661852 2629 maintenance_manager.cc:643] P c37183c2965f46a28847c1262eb97829: UndoDeltaBlockGCOp(8bf4675882e64f35bc3dc9ee06ae366d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 14:21:47.667095 26955 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:21:47.667645 26955 tablet_bootstrap.cc:492] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Bootstrap complete.
I20240503 14:21:47.667982 26955 ts_tablet_manager.cc:1367] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240503 14:21:47.668882 26955 raft_consensus.cc:348] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:47.669211 26955 raft_consensus.cc:729] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685c6181561b4056abdecfd6d8bdb3b2, State: Initialized, Role: FOLLOWER
I20240503 14:21:47.669543 26955 consensus_queue.cc:260] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "285488446ac445c5a5dfe654bfaf9835" member_type: VOTER last_known_addr { host: "127.20.99.6" port: 42763 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "685c6181561b4056abdecfd6d8bdb3b2" member_type: VOTER last_known_addr { host: "127.20.99.2" port: 43457 } }
I20240503 14:21:47.670449 26955 ts_tablet_manager.cc:1398] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:21:47.671510 4355 tablet_copy_service.cc:342] P 285488446ac445c5a5dfe654bfaf9835: Request end of tablet copy session 685c6181561b4056abdecfd6d8bdb3b2-32927a23842b4725b9dfafbac47fe754 received from {username='slave'} at 127.20.99.2:37851
I20240503 14:21:47.671644 4355 tablet_copy_service.cc:433] P 285488446ac445c5a5dfe654bfaf9835: ending tablet copy session 685c6181561b4056abdecfd6d8bdb3b2-32927a23842b4725b9dfafbac47fe754 on tablet 32927a23842b4725b9dfafbac47fe754 with peer 685c6181561b4056abdecfd6d8bdb3b2
I20240503 14:21:47.692265 2431 raft_consensus.cc:1203] T b359fb8d9afc4c9c856897a722100304 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240503 14:21:47.694805 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:21:47.718991 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:21:47.720273 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:47.733639 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:47.746815 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:47.765789 2478 maintenance_manager.cc:419] P 685c6181561b4056abdecfd6d8bdb3b2: Scheduling UndoDeltaBlockGCOp(32927a23842b4725b9dfafbac47fe754): 4923 bytes on disk
I20240503 14:21:47.766252 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:47.767227 2369 maintenance_manager.cc:643] P 685c6181561b4056abdecfd6d8bdb3b2: UndoDeltaBlockGCOp(32927a23842b4725b9dfafbac47fe754) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
W20240503 14:21:47.786484 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:21:47.796511 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:47.826169 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:47.950978 2692 raft_consensus.cc:1203] T 8bf4675882e64f35bc3dc9ee06ae366d P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240503 14:21:47.989899 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:48.005038 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:48.014364 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:48.035440 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:48.035488 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:48.041642 2431 raft_consensus.cc:1203] T 32927a23842b4725b9dfafbac47fe754 P 685c6181561b4056abdecfd6d8bdb3b2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240503 14:21:48.057868 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:21:48.069092 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:48.090840 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:21:48.211467 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:48.223512 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:48.231809 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:48.241197 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:48.252727 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:48.278470 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:48.283290 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 14:21:48.344514 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:21:48.502386 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:48.503576 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:21:48.611459 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 14:21:48.664304 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:48.665791 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:48.670588 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:48.693068 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:48.697530 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:48.755102 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:48.847084 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240503 14:21:48.867974 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:48.890353 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:48.899736 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:48.900127 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:48.923370 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:48.923848 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 14:21:48.924230 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:49.053581 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:49.104645 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 14:21:49.126986 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:49.132232 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:49.140147 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:21:49.140919 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:49.157783 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:49.178459 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:49.261384 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 14:21:49.337057 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:49.343457 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:49.366067 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:49.367136 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:49.371505 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:49.407464 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:49.486140 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:49.598361 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:49.618180 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:21:49.619071 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:49.622927 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:49.630167 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:49.650293 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:49.765134 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:49.847957 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:49.873350 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:49.874444 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:49.875499 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:49.876338 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:49.901468 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:50.037422 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:21:50.125145 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:21:50.132948 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:21:50.241816 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:21:50.254099 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:21:50.284087 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:50.290225 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:50.290568 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:50.314720 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:50.324276 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:50.326082 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:50.381089 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:50.512132 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:21:50.526608 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:50.546787 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:50.547984 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:50.553726 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:50.560787 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:50.637945 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:21:50.716380 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:50.765370 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:50.769652 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:50.795655 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:50.802912 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:21:50.813313 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:50.820466 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:50.827291 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:50.876034 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:21:50.927304 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240503 14:21:50.953182 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:51.052912 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240503 14:21:51.066215 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:51.073623 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:51.077594 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:51.080804 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:51.090550 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:51.126541 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:51.127863 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 14:21:51.254886 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 14:21:51.289060 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:51.296645 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:51.319525 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:51.333598 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:51.363858 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:51.366840 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 14:21:51.447227 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:51.503048 2144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 14:21:51.537164 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:51.574368 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 14:21:51.582006 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:51.588246 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:51.591475 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:51.611394 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:51.635057 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:51.705513 27559 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:51.705721 27559 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:51.706418 27559 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:51.707966 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:21:51.708106 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:51.708232 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:21:51.708467 27559 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:51.727155 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:21:51.883816 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:21:51.911761 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:51.925355 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 14:21:52.004285 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 14:21:52.010593 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:52.016789 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:52.030568 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:52.048082 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:52.055647 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:52.064657 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:52.071352 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:52.251765 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:52.272918 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:52.274248 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:21:52.278371 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:52.285099 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:52.304148 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:52.312695 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:52.488556 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:21:52.505200 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:52.518365 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:52.523902 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:52.544446 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:52.545615 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:21:52.548305 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:52.602208 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:52.728263 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:21:52.764077 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:52.771234 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:52.779343 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:52.784406 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:52.796593 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:52.802500 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:52.852088 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:52.983731 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:53.000216 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:53.006644 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:53.023588 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:53.027287 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:53.163244 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:53.225699 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:53.236253 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:53.237416 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:53.250254 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:53.253170 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:53.301227 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:21:53.302117 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:21:53.304073 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:53.335433 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240503 14:21:53.376542 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:21:53.446704 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:21:53.656020 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:53.668080 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:53.669697 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240503 14:21:53.680701 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240503 14:21:53.683421 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:53.685142 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:53.687695 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:53.694784 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:53.872725 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:53.892243 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:53.904601 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:53.907919 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:53.911612 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:53.936200 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:53.950090 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:21:54.037196 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240503 14:21:54.119707 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:54.136703 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:54.147579 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:54.147708 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:54.180990 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:54.203159 26107 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:54.203378 26107 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:54.204218 26107 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:54.205614 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:21:54.205827 2299 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:21:54.212007 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:54.221446 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:54.221588 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:54.221839 26107 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:54.223304 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 14:21:54.238941 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:54.257655 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:54.296247 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:54.364639 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:54.369017 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:54.369419 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:54.375958 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:54.399535 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:54.538244 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240503 14:21:54.627589 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:54.631821 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:54.642150 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:54.648952 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:54.652818 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:54.687017 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:21:54.811257 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:54.819293 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fda4460865854505b5220d7ec3b34c57: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:54.855356 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:54.891472 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:54.895531 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 14:21:54.902377 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:54.907491 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:54.911644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:54.938010 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:21:54.950325 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:55.297294 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:21:55.327380 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:55.335801 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:55.339229 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:55.350190 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:55.354720 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:55.361395 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:55.458240 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:21:55.463074 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:55.474226 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:21:55.537914 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:21:55.546878 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:55.553221 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:55.578476 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:55.588172 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:55.628885 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240503 14:21:55.635457 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:55.787434 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:21:55.818750 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:55.838546 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:55.849753 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:55.855691 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:55.871611 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:55.875447 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:55.909080 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:55.986979 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 14:21:56.071784 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:56.077783 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:56.095283 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:56.096698 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:56.099834 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:56.158402 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:21:56.175264 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240503 14:21:56.265373 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 14:21:56.297865 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:56.303225 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:56.325207 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:56.337123 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:56.341084 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:56.394820 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:56.502152 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:21:56.518587 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:56.580500 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:56.596977 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:56.598925 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:56.599694 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:56.603291 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:56.701328 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:21:56.800284 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 14:21:56.818181 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:21:56.846644 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:21:56.876423 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:57.012092 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:57.019315 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:57.032002 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:57.046959 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:57.086565 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:57.111382 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:57.139446 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:21:57.208464 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:21:57.257699 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:57.264178 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:57.269178 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:57.276940 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:57.307917 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:57.317124 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:57.329326 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:21:57.436909 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:21:57.508435 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:57.526540 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:57.529371 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:57.538307 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:57.551749 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:57.604197 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:57.715965 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 14:21:57.726420 27559 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:57.726624 27559 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:21:57.727392 27559 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:21:57.730124 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:57.736039 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:57.753974 27264 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:21:57.754175 27264 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:57.755064 27264 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:57.757068 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:57.758999 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:57.759429 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:21:57.759505 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:21:57.759735 27264 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:57.761381 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:57.761641 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:57.762030 27559 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:21:57.769440 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:57.776489 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:57.793345 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:57.808781 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:57.815794 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:21:57.829174 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:57.829849 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:21:57.931344 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240503 14:21:57.982908 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:58.001461 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:58.014308 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:58.019340 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:58.034232 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:58.036550 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:58.127148 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240503 14:21:58.233374 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:58.244360 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:58.246296 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:58.263120 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:58.272014 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:58.407734 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 14:21:58.412832 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:58.419982 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:21:58.420817 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:21:58.424607 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:58.517123 26107 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:21:58.517344 26107 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:21:58.518154 26107 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:21:58.518661 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:21:58.518893 2299 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:21:58.519663 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:58.519974 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:58.520071 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:21:58.520326 26107 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:21:58.550616 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:21:58.677273 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:58.677795 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:58.698403 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:58.705188 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:58.729267 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:58.880057 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:21:58.883397 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:58.917510 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:58.917913 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:58.927590 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:58.942315 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:58.942756 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:58.973560 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:21:59.110831 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:21:59.155781 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:59.169862 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:21:59.172096 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 14:21:59.180233 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:59.187467 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:59.190198 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:59.190537 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:21:59.412860 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:21:59.435338 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:59.439136 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:21:59.442422 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 14:21:59.455101 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:59.458127 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:59.479970 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:59.526309 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:21:59.658342 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:21:59.661983 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:21:59.676558 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:21:59.684039 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:21:59.687498 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:59.746562 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:21:59.811558 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:21:59.850200 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:21:59.908582 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:21:59.910848 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:21:59.926882 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:21:59.931474 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:21:59.951373 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:21:59.957386 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:21:59.976501 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:00.050223 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:00.292629 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:22:00.325174 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:22:00.343904 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:00.352268 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:00.355010 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:00.371433 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:00.376695 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:00.390012 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:00.521162 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:00.580188 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240503 14:22:00.598279 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:00.603839 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:00.606770 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:00.611094 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:00.611272 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:00.633477 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240503 14:22:00.634778 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 14:22:00.810770 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:00.841109 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:00.842614 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:00.843381 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:00.864308 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:00.922600 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:00.935940 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 14:22:00.957810 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 14:22:00.958868 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f631c7713d24f4caf7550bb05147615: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:00.990952 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:01.016256 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:22:01.040303 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:01.058660 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:01.064663 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:01.066597 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:01.067009 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:01.080436 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240503 14:22:01.243116 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:01.245844 28817 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:01.246071 28817 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:01.247215 28817 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:22:01.248736 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:01.260417 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:01.260584 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:01.260872 28817 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:01.262439 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:01.284797 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:01.285744 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:01.308754 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:01.310602 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:01.317350 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:01.468466 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:22:01.535879 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:01.566061 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:01.574519 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:01.578785 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:01.587278 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240503 14:22:01.589125 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:01.611168 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:01.657883 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:22:01.670078 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 14:22:01.707897 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 14:22:01.765377 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 14:22:01.855742 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:01.983296 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:01.990176 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:02.003209 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:02.035429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:02.036504 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:02.135522 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:02.228734 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:02.228893 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:02.231844 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:02.249481 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:02.249835 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:02.268836 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:02.297700 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:22:02.298678 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:22:02.304353 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:02.385841 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 14:22:02.492887 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:02.503821 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:02.523496 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:02.532950 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:02.539733 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:02.679728 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:02.713341 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:02.750140 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:02.750231 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:02.774214 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:02.774645 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:02.778095 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:02.778470 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:02.900365 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:02.969399 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:02.989046 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:03.002749 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:03.017969 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:22:03.023739 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:03.027813 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:03.082391 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 14:22:03.107049 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:03.117280 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240503 14:22:03.134559 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:03.208117 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:03.211125 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:03.222880 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:03.228159 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:03.235894 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:03.243923 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:03.359072 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 14:22:03.465464 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 14:22:03.638656 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:22:03.639974 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:03.654453 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:03.662060 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:03.685075 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:03.686208 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:03.692467 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:03.707484 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:03.743624 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:03.767135 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:03.890661 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:03.896436 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:03.913385 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:03.920140 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:03.937325 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:04.135367 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:04.142947 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:04.157274 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:04.159219 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:04.162201 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:04.179844 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 14:22:04.190191 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 14:22:04.242287 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:04.265540 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:04.379736 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:04.402829 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:04.403996 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:04.404285 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:04.419253 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:04.440713 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:22:04.481959 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 14:22:04.620612 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:04.629949 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:04.645118 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:04.659451 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:04.660270 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:04.753434 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:22:04.774035 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:04.783409 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:04.834388 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:22:04.886781 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:04.887522 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:04.902292 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:04.904155 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:04.909631 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:05.087896 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:22:05.141564 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:22:05.225894 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:05.321209 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:05.330734 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:05.331269 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:05.336961 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:05.339883 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:05.355657 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:05.504982 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:22:05.525434 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:05.542412 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:05.547627 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:05.558202 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:05.572089 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:05.575935 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:22:05.592324 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:22:05.712074 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:05.780191 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:05.784013 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:05.802605 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:05.808619 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:05.821382 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:05.832897 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 14:22:05.834736 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240503 14:22:05.862444 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:22:05.867995 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:05.967578 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 34 similar messages]
I20240503 14:22:06.022537 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:06.023223 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:06.027907 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:06.037436 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:06.067126 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:06.137467 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:22:06.156103 29367 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:06.156265 29367 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:06.157138 29367 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:06.158186 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:22:06.158412 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:22:06.158905 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:06.159232 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:06.159332 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:06.159574 29367 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:06.174803 29293 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:06.175032 29293 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:06.175716 29293 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:06.177731 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:06.177850 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:06.177937 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:06.178258 29293 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:06.178449 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:06.203028 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:22:06.249729 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:06.253593 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:06.270552 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:06.278280 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:06.298414 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:06.302834 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:06.518779 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240503 14:22:06.533104 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:06.543663 29367 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:06.543856 29367 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:06.544731 29367 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:22:06.547762 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:06.548569 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:06.548671 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:06.548880 29367 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:06.548986 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:06.552955 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:06.570303 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:06.576062 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:06.594607 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:06.643520 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:06.647305 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:22:06.648346 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 14:22:06.693379 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:06.870683 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 14:22:06.992029 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:07.029598 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:07.030236 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:07.038509 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:07.054167 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:07.122604 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:22:07.200137 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:07.230357 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:07.237937 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:07.243427 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:07.245446 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:07.249351 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:07.267396 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:07.297865 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:22:07.298702 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:22:07.351613 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:22:07.473209 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:07.500420 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:07.500597 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:07.501969 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:07.502635 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:07.607280 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:22:07.644979 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:07.662710 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:22:07.666602 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:07.671308 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:22:07.713470 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:07.726342 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:07.745100 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:07.753602 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:07.767659 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:07.932565 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:07.944420 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:07.965958 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:07.975827 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:07.980098 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:08.025395 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 14:22:08.151849 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:22:08.171805 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:08.208154 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:08.210043 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:08.225356 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:08.228013 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:08.233099 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:08.234118 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:08.258530 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 14:22:08.280844 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240503 14:22:08.403460 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:22:08.438656 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:22:08.461990 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:22:08.488885 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 14:22:08.489967 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:08.618528 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:08.627275 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:08.627295 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:08.643034 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:08.649704 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:08.698760 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:22:08.700806 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:08.731421 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:08.837049 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:08.839257 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:08.852777 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:08.871971 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:08.872211 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:08.901778 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 14:22:09.079396 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:09.081640 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:09.094306 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:09.098613 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:09.107672 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:09.181694 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:09.217700 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:22:09.223623 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:09.317859 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:09.323556 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:09.326508 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:09.326973 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:09.341833 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:09.385501 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240503 14:22:09.536592 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:09.539216 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:09.551734 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:09.552017 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:09.580119 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:09.710107 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:22:09.733242 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:22:09.743260 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:22:09.766819 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:09.774765 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:09.780267 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:09.796015 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:09.798923 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:09.822474 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:09.824513 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:10.022557 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:22:10.189535 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:22:10.223980 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:10.236855 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:10.242137 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:10.264151 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:10.268362 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:10.271025 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:10.287160 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:10.411249 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:22:10.463241 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:10.475407 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:10.478618 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:22:10.485273 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:10.512924 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:10.542063 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:10.592314 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 14:22:10.659430 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240503 14:22:10.673780 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:10.708653 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:10.724351 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:10.724462 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:10.739775 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:10.752018 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:10.800659 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:10.885568 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:22:10.917214 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:10.921628 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:10.928000 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:10.938666 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:10.952175 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:10.967483 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:10.991014 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240503 14:22:10.991349 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:22:11.062812 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:11.120826 30189 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:11.121021 30189 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:11.121915 30189 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:11.122242 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:22:11.122470 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:22:11.123425 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:11.130024 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240503 14:22:11.130975 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:22:11.144485 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:11.144639 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:11.144989 30189 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:11.158887 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:11.166522 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:11.180567 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:11.187794 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:11.201946 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:11.219960 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:11.252240 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fda4460865854505b5220d7ec3b34c57: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:11.396762 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:11.411617 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240503 14:22:11.419031 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:11.424422 29367 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:11.424593 29367 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:11.425487 29367 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:22:11.428761 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:11.429019 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:11.429111 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:11.432546 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:11.445044 30417 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:11.447531 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:11.450668 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:11.470815 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:11.567677 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 14:22:11.667371 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:11.695727 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 14:22:11.724448 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:11.860486 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:11.862268 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:11.868624 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:11.872499 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:11.882436 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:12.097946 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:12.105262 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:12.106150 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:12.106179 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:12.116150 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:12.121079 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:12.159683 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:22:12.173936 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:22:12.326380 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:12.336249 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:12.337913 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:12.338928 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:12.339327 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:12.345906 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:12.365360 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:12.527024 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:12.559888 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:12.569053 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:12.573195 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:22:12.574235 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:22:12.580659 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:12.580642 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:12.617888 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:12.712432 30137 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:12.712649 30137 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:12.713322 30137 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:12.714843 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:22:12.716430 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:12.716564 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:12.717049 30137 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:12.720652 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:22:12.803915 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:12.824748 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:12.833868 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:12.850648 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:12.856083 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:12.924510 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:12.945040 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:22:12.977543 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:13.050769 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:13.067284 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:13.080343 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:13.081288 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:13.082888 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:13.250270 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:22:13.255527 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 14:22:13.261472 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:22:13.294080 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:22:13.422295 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 14:22:13.429157 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:22:13.473974 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240503 14:22:13.476192 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:13.491925 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:13.498545 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:13.499531 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:13.502466 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:13.533828 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:13.543295 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:13.545879 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:22:13.757486 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:13.776214 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:13.780333 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:22:13.789717 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:13.802457 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:13.808589 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:13.876556 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:22:13.960520 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:14.006386 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:14.020572 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:14.026566 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:14.029518 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:14.032665 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:14.122282 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:14.230216 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:14.237325 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:14.242105 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:14.248097 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:14.281692 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:14.402595 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:14.444581 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:22:14.471489 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:14.491533 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:14.493728 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:14.496918 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:14.504002 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:14.534045 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:22:14.573194 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:22:14.578155 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:14.601403 30137 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:14.601575 30137 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:14.602300 30137 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:14.604477 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:14.606487 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:14.606599 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:14.606833 30137 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:14.670126 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:22:14.707684 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:14.735200 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:14.746771 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:14.747228 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:14.759399 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:14.936434 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:15.020506 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:22:15.154300 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:15.155231 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:15.168893 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:15.181536 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:15.186830 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:15.213932 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:15.336627 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:15.365412 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:15.370765 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:15.398962 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:15.400413 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:15.401257 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:15.403496 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:15.472106 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:22:15.596711 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:15.609127 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:15.622525 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:15.624881 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:22:15.638864 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:15.648424 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:15.668853 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:15.741811 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:15.812397 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:15.847124 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:22:15.848706 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:15.866382 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:15.875313 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:15.879593 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:15.895051 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:15.910569 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 14:22:15.969633 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:22:15.978929 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:22:16.088281 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:16.098539 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:16.104593 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240503 14:22:16.120946 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:22:16.121328 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:16.133924 30956 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:16.134132 30956 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:16.134609 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:16.134822 30956 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:16.138080 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:16.138484 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:16.138607 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:16.140671 30956 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:16.142199 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:16.162647 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:16.327919 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:16.364528 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:16.371822 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:16.386335 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:16.399597 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240503 14:22:16.413352 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:16.414045 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:16.446074 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240503 14:22:16.501852 31029 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:16.502038 31029 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:16.502787 31029 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:16.503218 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:22:16.503475 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:22:16.504487 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:16.504604 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:16.504716 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:16.504917 31029 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:16.626682 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:16.688984 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:22:16.782644 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:16.786116 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240503 14:22:16.814889 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:16.827267 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:16.831086 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:16.840376 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:16.859671 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:17.006803 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:22:17.030159 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:17.031255 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:17.038228 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:17.044281 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:17.045897 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:17.109983 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:17.173426 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:22:17.243108 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:17.255493 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:17.256721 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:17.263401 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:17.280705 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:17.281020 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:17.462240 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 14:22:17.488395 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:17.502393 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:17.504211 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:17.508391 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:17.509788 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:17.522425 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:17.625416 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 14:22:17.626312 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:22:17.712499 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:17.730226 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:17.747877 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:17.768353 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:17.779379 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:17.786324 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:17.819797 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 14:22:17.941239 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:17.981205 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:17.981799 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:17.994824 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:17.997902 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:17.999786 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:22:18.023492 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:18.136011 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:22:18.232611 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:18.246465 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 14:22:18.339470 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:18.424332 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240503 14:22:18.428857 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:18.440652 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:18.470372 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:18.481019 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:18.483007 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:18.491402 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240503 14:22:18.572737 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:22:18.643958 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 14:22:18.671886 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:18.674785 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:18.693190 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:18.697677 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:18.702207 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:18.777931 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:18.821894 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:18.831897 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:22:18.865149 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:22:18.900890 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:18.907486 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:18.911585 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:18.948275 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:18.987112 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:19.028762 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:22:19.149473 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:19.150851 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:19.178659 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:19.181388 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:19.187896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:19.239396 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:19.339851 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:19.342316 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:19.370138 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:19.380596 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:19.389695 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:19.407241 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:19.409684 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:19.411402 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:22:19.423306 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:19.560403 2144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 14:22:19.561463 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:19.617756 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:19.637223 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:19.639950 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:19.663910 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:19.665143 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:19.709057 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:22:19.723487 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:19.731480 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:22:19.888427 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:20.058221 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:20.063670 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:20.065160 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:20.066192 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:20.091991 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:20.126363 4269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 46851.6us
W20240503 14:22:20.184731 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:22:20.205014 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:20.275308 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:20.301079 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:20.301476 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:20.310962 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 14:22:20.318665 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:20.324577 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:20.475443 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:20.524873 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:22:20.543449 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:20.566581 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:20.566571 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:20.585464 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:20.590648 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:20.687620 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:20.703140 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:22:20.785378 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:20.799062 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:20.799504 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:20.801955 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:20.821547 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:20.905382 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240503 14:22:20.915264 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240503 14:22:20.926445 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:22:20.987725 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:21.006760 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:21.017328 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:21.017611 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:21.021565 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:21.050822 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:21.146137 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:22:21.196321 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:21.249163 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:21.255260 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:21.276441 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:21.285012 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:21.322001 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:21.350293 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:22:21.485672 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:22:21.486554 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:21.495249 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:21.672096 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:21.694972 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:21.713950 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:21.714388 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:21.721863 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:21.740950 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:21.789747 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240503 14:22:21.791548 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:22:21.849381 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240503 14:22:21.935969 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:21.946408 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:21.992146 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:22:21.994597 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:22.001711 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:22.002621 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:22.019095 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:22.021253 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 14:22:22.071488 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:22.075096 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:22.135828 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:22:22.188854 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:22.198978 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:22.216814 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:22.226722 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:22.244607 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:22.439627 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:22.443351 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:22.443759 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:22.452822 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:22.461071 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:22.466864 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:22.502503 31655 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:22.502758 31655 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:22.503726 31655 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:22.505951 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:22.539314 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:22.542791 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:22.542941 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:22.543165 31655 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:22.547456 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 14:22:22.571102 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:22:22.672019 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:22.691666 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:22.692291 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:22.699661 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:22.718737 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:22.728044 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 14:22:22.925962 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:22.928990 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:22.940764 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:22.958058 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:22.959722 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:22.975685 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:23.056223 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:23.087256 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:23.102131 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:23.332058 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240503 14:22:23.400650 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:23.415115 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240503 14:22:23.425379 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:23.451524 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:23.476876 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:23.479694 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:23.500602 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 14:22:23.536474 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:23.561940 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:23.647440 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:23.669406 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:23.680946 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:23.688932 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:23.694823 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:23.752235 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:23.782296 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:22:23.885462 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:23.901743 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:23.916779 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:23.917440 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240503 14:22:23.927397 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:23.933555 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:23.940285 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:24.081743 31655 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:24.081940 31655 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:24.082659 31655 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:24.086043 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:24.087108 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:24.087245 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:24.087492 31655 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:24.112752 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:24.121088 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:24.125442 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:24.148137 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:24.158267 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 14:22:24.159101 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:24.159310 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:24.165544 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:22:24.279281 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240503 14:22:24.356065 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:24.356906 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:24.357686 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:24.358247 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:24.358817 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:24.422804 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:22:24.446100 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:22:24.479848 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:24.600632 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:22:24.617427 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:24.626490 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:24.641701 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:24.642652 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:24.657553 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:24.662037 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:24.967371 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:25.035591 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:25.038816 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:25.052832 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:25.059268 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:25.065093 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:25.118822 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:22:25.159636 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:22:25.233660 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:22:25.255836 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:25.289402 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:25.303992 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:25.310225 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:25.322019 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:25.327109 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:25.386534 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:25.489846 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240503 14:22:25.510772 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:22:25.534329 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:25.536765 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:25.561314 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:25.575639 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:25.589082 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:22:25.626379 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:25.724515 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:25.789606 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:25.799530 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:25.803460 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:25.814445 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:25.822911 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:26.000595 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:26.003705 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:26.011279 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:26.014345 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:26.016904 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:26.038664 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:26.074517 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240503 14:22:26.136793 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:22:26.196822 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:22:26.208974 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:22:26.263243 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:26.277451 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:26.290400 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:26.291110 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:26.316701 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:26.317241 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:26.410363 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:22:26.498919 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:26.694726 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:26.714833 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:22:26.715126 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240503 14:22:26.718118 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:26.719489 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:26.725898 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:26.735420 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:26.739652 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:22:26.751470 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:26.816316 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 14:22:26.858994 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 14:22:26.860102 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:22:26.907969 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:26.914283 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:26.932812 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:26.949421 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:26.984427 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:26.998901 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:27.139459 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:27.170020 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:27.181396 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:27.196997 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:27.206992 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:27.235365 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:27.275307 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:22:27.412662 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:27.435343 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:27.460929 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:27.468569 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:27.469723 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:27.497658 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:27.610543 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:27.635246 31029 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:27.635504 31029 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:27.636833 31029 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:27.637873 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:22:27.638185 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:22:27.641839 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:27.646209 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:22:27.649482 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:27.649645 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:27.649950 31029 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:27.679248 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:27.697232 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:27.703912 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:27.721258 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:27.738130 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:27.773417 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:22:27.824609 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:22:27.864048 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:22:27.884433 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:22:27.928061 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:22:27.961205 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:27.975131 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:27.998684 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:27.998935 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:28.046149 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:22:28.047374 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:28.056727 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:28.225836 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:28.405807 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:28.420264 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:28.444067 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:28.502566 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:28.514757 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:28.515992 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:28.642083 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:28.683158 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:28.684383 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:28.689584 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:28.700438 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:28.721436 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240503 14:22:28.722756 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:28.727892 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 14:22:28.740062 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:22:28.810096 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240503 14:22:28.906307 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:28.911438 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:28.912346 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:28.940151 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:28.951740 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:29.017395 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:22:29.019251 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:29.103032 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:22:29.156067 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:29.164423 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:29.178047 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:22:29.182060 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:29.183331 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:29.188997 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:29.217020 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:29.257342 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:22:29.263446 2498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.6:42763, user_credentials={real_user=slave}} blocked reactor thread for 52059.4us
I20240503 14:22:29.406685 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:29.432020 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:29.435427 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:29.446823 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:29.463158 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:29.497804 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:29.582233 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:29.625042 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:22:29.645921 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:29.659358 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:29.674429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:29.688679 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:29.699690 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:29.727599 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:29.860867 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:29.897310 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:30.108769 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:30.118417 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:30.123538 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:30.128940 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:30.150127 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:30.223963 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:30.240489 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:30.287460 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:22:30.337635 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:30.346159 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:30.360180 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:30.363332 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:30.367614 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:30.413305 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:30.413429 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:30.423377 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:22:30.476055 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:30.576656 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:30.586344 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 14:22:30.593796 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:30.599740 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:30.600589 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:30.624945 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:30.667309 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:30.823427 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:30.841888 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:30.842511 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:30.843966 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:30.856770 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:31.026599 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:22:31.040614 31029 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:31.040884 31029 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:31.042021 31029 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:22:31.045783 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:22:31.045938 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:31.046092 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:31.046451 31029 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:31.047407 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:31.057334 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:31.064203 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:31.070179 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:31.080421 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:31.093210 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:31.134585 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:31.146956 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240503 14:22:31.197510 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:31.203155 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 14:22:31.227198 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 14:22:31.302758 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:31.314601 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:31.323501 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:31.337144 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:31.342662 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:31.511890 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:22:31.587276 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:22:31.614717 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:31.687954 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:31.725149 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:31.776306 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:31.792295 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:31.798681 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:31.801002 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:31.803802 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:31.868839 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:31.939479 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20240503 14:22:31.966475 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240503 14:22:31.983146 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:31.988848 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:31.999644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:32.004372 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:32.008162 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:32.015565 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:32.069994 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:22:32.191382 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:32.229205 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:32.231146 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:32.243425 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:32.243820 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:32.313987 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:32.407728 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:32.434461 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:22:32.456413 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:32.456748 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:32.475301 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:32.493813 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:32.497458 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:32.689404 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:32.697842 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:32.700484 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:32.703001 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:32.710704 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:32.714905 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:22:32.733876 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:32.738314 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:22:32.825774 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:32.827888 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:32.955633 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:32.956996 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:32.966986 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:32.972533 31029 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:32.972689 31029 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:32.973477 31029 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:32.974475 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:22:32.974679 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:22:32.976578 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:32.979456 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:32.980540 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:32.997221 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:32.997380 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:32.997905 31029 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:33.024015 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:22:33.064831 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:22:33.136548 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 14:22:33.263489 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:33.285637 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 14:22:33.356760 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:33.371289 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:33.373319 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:33.376220 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:33.404774 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:33.411249 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:33.436939 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:22:33.595028 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:33.612545 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:33.618708 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:33.621421 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:33.658469 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:33.704136 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:22:33.712694 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 14:22:33.761703 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:33.771697 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 14:22:33.772747 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f631c7713d24f4caf7550bb05147615: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:33.833709 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:33.835227 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:33.841099 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 14:22:33.852277 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:33.856600 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:33.857074 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:33.858114 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:33.939617 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240503 14:22:34.033131 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:22:34.049242 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:34.068631 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:34.069072 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:34.083837 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 14:22:34.087544 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:34.088358 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:34.113569 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:22:34.197726 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:34.272133 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:34.302426 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:22:34.302628 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:34.302685 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:34.305702 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:34.371840 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:34.396704 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:34.535895 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:34.552081 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:34.552943 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:34.563869 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:34.613777 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:34.635341 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:22:34.656690 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:34.844054 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:35.015125 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:35.020130 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:35.043998 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:22:35.052544 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:35.052564 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:35.054308 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:35.097385 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:35.153574 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:22:35.193141 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 14:22:35.246481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:35.264894 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:35.276439 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:35.280121 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:35.293190 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:35.295761 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:22:35.311868 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:35.370952 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:22:35.426162 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:35.498839 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:35.516669 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:35.527436 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:35.538987 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:35.540004 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:35.564626 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:22:35.717036 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 14:22:35.728699 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:35.746304 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:35.767633 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:35.779453 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:35.782245 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:35.844097 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:22:35.904662 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:35.953485 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:35.970983 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:35.982720 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:35.983851 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:35.985563 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:36.029042 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:36.142429 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240503 14:22:36.177268 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:36.193127 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:36.201385 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:36.207231 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:36.212571 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:36.336997 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 14:22:36.377331 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:36.424892 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:22:36.430390 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240503 14:22:36.446717 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:22:36.465795 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240503 14:22:36.495316 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:36.524947 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 14:22:36.630961 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:36.649667 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:36.661056 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:36.671005 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:36.676653 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:36.702201 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:22:36.878275 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:36.885169 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:36.897755 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:36.920732 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:36.921535 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:36.968948 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 14:22:36.969853 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:36.981528 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:37.135228 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:37.143316 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:37.159593 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:22:37.164285 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:37.166608 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:37.179234 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:37.367960 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:37.385694 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:37.387888 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:37.397534 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:37.415226 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:37.416988 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:37.437141 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:22:37.527271 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:37.609920 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:37.623821 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:37.644191 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:37.644482 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:22:37.645606 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:37.664618 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:22:37.665639 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:22:37.666273 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:37.775653 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:37.852967 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:37.863229 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:37.871794 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:37.892290 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:37.938992 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:37.972033 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:38.036222 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 14:22:38.057909 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:22:38.135578 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240503 14:22:38.320845 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:38.341019 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:38.346784 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:38.349421 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:38.358366 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:38.373245 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:38.395318 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 14:22:38.451797 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:22:38.454525 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:38.546211 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:38.550983 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:38.554910 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:38.555322 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:38.555737 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:38.632794 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 14:22:38.699767 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 14:22:38.777874 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:38.790597 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:38.795348 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:38.810554 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:38.817829 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:38.837647 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:38.940075 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:22:38.964439 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:38.977561 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 14:22:39.001772 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:39.003010 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:39.010164 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:39.010502 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:39.015889 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:39.075060 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:22:39.145980 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:22:39.193630 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:39.206085 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:39.208873 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:39.229279 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:39.232470 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 14:22:39.237874 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:39.257258 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240503 14:22:39.440929 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:39.447151 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:39.454876 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:39.476111 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:39.476630 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:39.479998 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:39.516646 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:39.643352 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:22:39.685724 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:22:39.865850 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:39.877647 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:39.884008 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:39.886329 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:39.893929 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:39.903512 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:40.057490 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:40.072696 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:22:40.088470 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:40.092109 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:40.098639 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:40.102238 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:40.104938 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:40.176599 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:40.247066 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:22:40.312431 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:40.331820 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:40.344233 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:40.346441 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:40.352375 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:40.355967 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:40.446857 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:40.562940 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:40.570523 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:40.572932 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:40.597764 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240503 14:22:40.609328 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:40.616806 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:40.639593 2496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 61626.6us
W20240503 14:22:40.702354 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:40.714593 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:22:40.798501 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:40.805830 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:40.813239 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:40.815279 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:40.832746 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:40.923468 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 14:22:41.001251 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:41.006911 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:41.019820 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240503 14:22:41.026196 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:41.029196 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:22:41.030481 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:41.032927 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:41.052541 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:41.074154 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 14:22:41.241518 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:41.296063 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 14:22:41.297179 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:41.427318 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:41.443315 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:41.454629 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:41.454712 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:41.457531 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:41.478770 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:41.496935 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:41.570487 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240503 14:22:41.595739 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240503 14:22:41.647467 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:41.666270 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:41.672143 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:41.677978 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:41.691843 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:41.712328 1696 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:41.712508 1696 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:41.713402 1696 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:41.714279 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:22:41.714522 2299 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:22:41.715764 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:41.716485 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:41.716773 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:41.716882 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:41.717299 1696 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:41.747879 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:22:41.848949 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:22:41.913057 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:41.917575 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240503 14:22:41.927388 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:41.932320 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:41.932875 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:41.933252 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:41.984850 2233 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:41.985021 2233 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:41.993036 2233 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:41.997303 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:42.001935 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:42.002084 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:42.002343 2233 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:42.014340 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:42.066345 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:42.086665 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:22:42.131636 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:42.158404 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:42.172530 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:42.172739 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:42.178838 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:42.185814 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:42.254195 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:22:42.383929 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:42.400183 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:42.411764 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:42.415848 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:42.418040 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:42.445775 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:42.603390 2233 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:42.603612 2233 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:42.604398 2233 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:42.606038 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:42.609150 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:42.609297 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:42.609570 2233 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:42.623813 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:42.635628 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:22:42.636749 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:42.641853 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:42.646879 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:42.653904 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:42.666208 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:22:42.668151 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:42.821524 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:42.988633 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:22:43.052487 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:43.057228 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:43.057670 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:43.059873 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 44 similar messages]
I20240503 14:22:43.062171 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:43.079142 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:43.152423 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:43.251037 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:43.257735 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:43.267766 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:22:43.275993 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:43.279335 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:43.283726 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:43.303887 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 14:22:43.310077 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:22:43.315860 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:43.316617 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:43.389129 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 14:22:43.490582 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:43.496353 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:43.514822 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:43.517716 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:43.517886 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:43.545536 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 14:22:43.561378 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:43.717519 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:43.723824 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:43.729707 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:43.737969 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:43.747356 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:43.754887 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:43.970897 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:43.979271 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:43.985822 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:44.000298 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:44.007444 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:44.010668 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 14:22:44.016361 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:44.032150 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240503 14:22:44.056835 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fe0eebf47c494dad8edf06197cdf15bc: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:44.064929 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:22:44.177517 2233 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:44.177726 2233 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:44.178419 2233 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:44.180038 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:22:44.180338 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:44.180461 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:44.180869 2233 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:44.191887 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:44.216226 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:44.218500 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:44.223464 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:44.230029 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:44.243856 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:44.513005 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:22:44.537447 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:44.590150 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 14:22:44.599402 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:22:44.616832 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:44.625521 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:44.651731 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:44.666888 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:44.666903 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:44.682747 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:44.686846 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:44.873418 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:44.891436 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:44.899578 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:44.919497 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:44.920079 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:44.972548 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:22:44.972983 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:22:45.099998 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:45.106729 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:45.114746 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:45.118009 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:45.135049 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:45.135437 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:22:45.160511 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:45.214493 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:45.328310 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:45.336242 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:22:45.351089 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:45.359529 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:45.360842 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:45.362473 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:45.491055 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 14:22:45.573844 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:22:45.577556 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:45.582304 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:45.598773 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:45.611151 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:45.617529 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:45.664366 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:45.681438 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:45.775465 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 14:22:45.823582 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 14:22:45.833595 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:45.847288 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:45.847797 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:45.861699 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:45.879918 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:45.899531 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:22:46.076716 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 14:22:46.112208 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:46.267050 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:22:46.274495 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:46.323259 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:46.343848 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:46.344018 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:46.359692 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:46.368835 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:46.419272 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240503 14:22:46.498673 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:46.595038 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:46.619577 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:46.620208 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:46.625156 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:46.625681 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:46.684978 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 14:22:46.819700 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:46.823915 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:46.836968 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:46.846213 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:46.860507 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:46.866760 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:46.913394 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 14:22:47.069486 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:22:47.081745 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:47.083560 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:47.083806 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:47.089200 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:47.098171 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:22:47.099094 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:47.105316 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:22:47.106051 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:47.193706 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:22:47.264046 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 14:22:47.288877 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:47.304945 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:47.311479 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:47.316035 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:47.349299 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:47.364876 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:47.492355 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:22:47.539252 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:47.544466 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:22:47.555395 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:47.566370 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:47.573886 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:47.578694 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:47.668486 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:47.689918 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:22:47.936867 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:48.010457 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:48.023440 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:48.027348 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:48.028173 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:48.051223 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:48.064342 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:22:48.067420 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:48.106642 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 14:22:48.144172 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:22:48.169469 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 14:22:48.204972 3546 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:48.205161 3546 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:48.205948 3546 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:22:48.208055 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:48.208385 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:48.208480 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:48.209118 3546 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:48.209843 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:48.222674 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:48.248284 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:48.249019 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:48.272634 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:48.279740 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:48.395375 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:48.396828 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:22:48.445641 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 14:22:48.477371 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:48.477782 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:48.493381 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:48.498576 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:48.502635 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:48.521595 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:48.583206 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240503 14:22:48.730865 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:48.739621 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:48.753243 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:48.762180 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:48.803905 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:48.934518 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240503 14:22:48.935225 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:48.959211 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:48.976281 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:48.977056 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:22:48.978459 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:48.980070 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:48.999002 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:49.034247 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:49.169108 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:49.186789 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:49.191535 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:49.197201 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:49.202728 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:49.304666 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 14:22:49.382401 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:49.475847 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:22:49.584990 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:49.600315 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 14:22:49.606112 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:49.622516 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:49.629765 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:49.636240 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:49.667811 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:49.737730 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:22:49.777065 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:22:49.878300 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:49.879166 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:49.891440 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:49.896256 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:22:49.906425 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:49.920490 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:49.920691 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:50.048468 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:22:50.095400 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:50.103940 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:50.116114 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:50.124630 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:50.136818 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:50.140218 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:50.156383 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:22:50.261193 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:22:50.310194 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:50.320217 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:50.330049 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:50.333494 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:50.340293 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:50.353152 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:50.539521 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:50.543527 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:50.547583 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:50.564575 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:50.567170 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:50.594316 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:22:50.738665 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:50.764701 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:50.769796 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:50.769920 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:50.780153 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:50.785710 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:50.823096 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 14:22:50.823824 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:22:50.866257 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 14:22:50.874650 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 14:22:50.894280 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:51.133561 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 14:22:51.157267 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:51.191306 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:51.220571 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:51.222543 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:51.223172 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:51.241952 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:51.292529 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:51.367676 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:22:51.408192 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240503 14:22:51.432906 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:51.445801 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:51.469870 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:51.476498 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:51.505746 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:51.670044 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:51.675540 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:51.690272 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:51.692344 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:51.715724 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:51.723603 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:51.748980 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:51.763517 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:51.797451 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 14:22:51.901073 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:51.922799 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:51.926477 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:51.933750 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:51.944731 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:52.113723 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:52.120528 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:52.120815 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:52.131489 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:52.139684 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:52.155661 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:52.199443 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:22:52.202405 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 14:22:52.272565 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:52.304656 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:22:52.305413 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:22:52.346675 2144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20240503 14:22:52.352072 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:52.361063 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:22:52.365350 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:52.368196 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:52.372470 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:52.383790 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:52.387537 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:52.425230 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:22:52.669716 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:52.726933 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:22:52.727062 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:22:52.789109 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:52.790722 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:22:52.802361 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:52.813999 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:52.814429 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:52.819070 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:52.874569 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:52.992484 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:53.003650 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:53.007375 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:53.019270 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:53.040403 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:53.170993 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:22:53.201015 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 14:22:53.205588 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:53.218757 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:53.231369 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:53.232501 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:53.232833 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:53.234992 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:53.262121 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240503 14:22:53.415459 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:22:53.447460 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:53.473626 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:53.476568 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:53.482067 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:53.488888 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 14:22:53.491277 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:53.508247 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:53.688485 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:53.706439 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:53.706522 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:53.709787 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:53.713640 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:53.718662 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:53.779139 4217 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:53.779346 4217 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:53.780038 4217 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:53.782521 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:53.783957 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:53.784075 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:53.784312 4217 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:53.798803 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240503 14:22:53.869957 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:22:53.889580 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240503 14:22:53.921523 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:53.932544 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:53.952781 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:53.959867 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:54.007584 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:54.014317 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:54.192000 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:54.197002 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 14:22:54.283922 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:22:54.384475 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:54.390103 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:54.407910 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:54.409399 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:54.477766 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:54.553004 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:54.569518 4737 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:22:54.569738 4737 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:54.570643 4737 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:22:54.571821 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:22:54.572070 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:22:54.572595 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:54.573314 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:54.573421 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:22:54.574823 4737 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:54.629184 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:54.632887 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:54.639207 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 14:22:54.650089 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:54.654600 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:54.655090 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:54.691306 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:54.845248 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:22:54.848492 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:54.848825 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:54.849804 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:54.858891 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:54.895006 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:54.931151 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:22:54.938668 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:22:55.064739 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:55.070856 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:22:55.080631 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:55.096592 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:55.098836 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:55.107606 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:22:55.110070 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:55.142836 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:55.293434 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:55.306094 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:55.317929 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:55.339843 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:55.355675 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:55.414819 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:55.452185 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:22:55.532860 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 14:22:55.579926 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:55.580736 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:55.595623 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:55.595790 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:22:55.596961 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:55.600279 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:55.601187 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 14:22:55.623034 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:55.855414 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240503 14:22:55.891006 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:56.022262 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:56.029577 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:56.042826 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:22:56.047459 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:56.065598 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:56.067826 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:56.084098 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:56.249339 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:56.257409 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:56.258450 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:56.263644 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:56.269165 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240503 14:22:56.271886 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:56.282413 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:56.361119 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 14:22:56.495469 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:56.509198 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:56.514019 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:56.528113 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:56.528940 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:56.593114 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:22:56.656772 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:56.711869 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:56.712524 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:56.735729 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:56.747725 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:56.751770 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:56.764348 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 14:22:56.771201 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:56.918349 4910 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:56.918598 4910 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:22:56.919983 4910 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:22:56.921492 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:22:56.922338 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:56.922472 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
W20240503 14:22:56.922578 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:56.922912 4910 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:22:56.967514 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:56.973126 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:56.973181 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:56.976410 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 14:22:56.982257 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:56.992543 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 14:22:57.000430 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:57.039048 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:57.139225 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:57.200474 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:57.206934 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:57.208662 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:57.210758 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:57.213433 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:22:57.278852 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:22:57.296242 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 14:22:57.340353 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:22:57.485924 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:22:57.568210 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:57.589944 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:22:57.607228 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:57.608238 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:57.620705 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:57.623637 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:57.638468 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:57.639322 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:57.747300 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:22:57.846297 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:57.850975 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:57.854137 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:57.901271 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:57.935498 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:57.957909 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:22:58.024829 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:58.066380 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:58.095633 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:58.112110 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:58.121564 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:58.137090 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:58.150245 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:58.172299 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 14:22:58.207809 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:22:58.212591 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 14:22:58.268651 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:22:58.293815 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 14:22:58.347961 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:58.354892 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:58.355451 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:58.361387 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:58.379475 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:58.511816 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:58.559573 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:58.580828 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:58.582602 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:22:58.584169 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240503 14:22:58.584880 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:58.594846 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:58.595542 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:58.647357 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:58.789803 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:58.794673 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:58.806015 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:58.810545 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:58.827162 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:58.859381 4217 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:22:58.859565 4217 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:22:58.860251 4217 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:22:58.869361 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:22:58.873467 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:58.873628 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:22:58.875226 4217 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:22:58.939133 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 14:22:59.057060 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:22:59.093098 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:22:59.193940 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:22:59.216118 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:59.228699 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:59.229403 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:59.240242 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:59.253198 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:22:59.433869 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:59.433984 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:59.447121 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:59.477375 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:22:59.481810 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:22:59.487000 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 14:22:59.495705 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:22:59.511557 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 14:22:59.610878 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:22:59.626506 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:22:59.667572 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:22:59.671975 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:22:59.674410 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:59.679502 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:59.687929 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:59.766607 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:22:59.915275 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:22:59.935439 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:22:59.936942 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:22:59.939684 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:22:59.945803 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:22:59.955924 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:22:59.956926 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:00.013016 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:00.032177 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:00.087217 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:23:00.130602 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:00.142145 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:00.145787 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:00.146142 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:00.159657 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:00.291625 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:23:00.366654 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:00.382092 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:00.385594 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:00.387760 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:00.413059 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:00.426028 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:00.430533 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:23:00.604683 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:00.664180 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 14:23:00.684854 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240503 14:23:00.798317 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 14:23:00.839605 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:00.842213 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:00.849471 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:00.863389 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:00.864630 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:00.870230 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240503 14:23:00.899047 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:00.969535 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:23:01.032508 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:01.037569 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:01.047052 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:01.068291 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:01.074247 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:01.087337 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:23:01.121762 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:01.279332 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:01.279418 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:01.305037 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:01.306317 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:01.328856 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:01.381749 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:01.390949 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 14:23:01.503316 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:01.517022 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:01.524952 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:01.553475 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:01.554080 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:01.616315 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:01.638778 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:23:01.747438 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:01.750267 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:01.763715 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:01.770347 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:01.784648 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:01.800233 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:01.972492 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:01.980206 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:01.983026 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:01.993089 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:02.004873 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:02.056079 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:02.069782 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 14:23:02.088104 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:23:02.117657 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 14:23:02.124987 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:23:02.352795 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 14:23:02.354094 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:02.400159 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:02.426200 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:02.444366 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:02.454159 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:23:02.456779 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:02.479894 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:02.509425 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:23:02.518405 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240503 14:23:02.559523 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:02.626929 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:23:02.681814 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:02.686115 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:02.715063 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:02.725677 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:02.727872 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:02.866756 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:02.929504 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:02.933351 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:02.940374 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:02.941926 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:02.952864 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:02.961921 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 14:23:03.003556 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 14:23:03.132429 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:03.186134 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:03.215454 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:03.216265 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:03.216648 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:03.219341 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:23:03.222956 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:23:03.233310 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:03.256088 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:03.402670 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:03.425680 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:03.427999 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:03.446139 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:03.462159 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:03.466598 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 14:23:03.470846 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:23:03.510895 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:03.557509 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240503 14:23:03.637188 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:03.645879 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:03.650573 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:03.651551 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:03.651609 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:03.677774 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:23:03.728320 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240503 14:23:03.796188 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:04.061560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:04.062155 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:04.075248 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:04.076996 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:04.083775 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:04.103932 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:23:04.120230 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:04.147648 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:23:04.236256 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:04.276365 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:04.290788 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:04.298437 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:04.300276 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:04.328840 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:04.534049 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 14:23:04.555523 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:04.560859 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:04.584722 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:04.598533 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:04.599956 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:04.602852 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:04.629743 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 14:23:04.670244 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:04.679939 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:23:04.798177 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:04.815649 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:04.820781 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:04.824170 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:04.827972 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:04.927039 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:23:05.009061 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:05.013015 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:05.026762 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:23:05.029695 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:05.036020 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:05.052850 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:05.110520 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:23:05.130164 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 14:23:05.131063 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:05.177181 6232 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:05.177381 6232 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:05.178375 6232 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:05.178622 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:23:05.178781 2299 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:23:05.179962 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:05.181353 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:05.181478 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:05.181921 6232 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:05.231884 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:05.248440 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:05.263754 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:05.267856 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:05.271972 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:05.281775 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:05.331874 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240503 14:23:05.516989 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:05.568311 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:23:05.698874 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:05.700868 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:05.704535 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:05.705096 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:05.712661 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:05.714637 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:05.770588 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:23:05.890985 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:05.907733 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:05.909231 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:05.911458 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:05.932598 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:05.975684 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:06.014039 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:23:06.115726 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:06.129957 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:06.130913 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:06.134366 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 14:23:06.141600 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:06.169445 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:06.186443 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:06.207600 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:23:06.283524 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240503 14:23:06.294924 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:23:06.376081 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:06.377352 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:06.400130 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:06.401489 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:06.402889 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:06.436151 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:06.637444 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:06.659665 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:06.662878 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:06.682914 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:06.701996 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:06.721199 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:23:06.787802 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:06.826732 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:06.865587 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:06.874781 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:06.884128 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:06.896709 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:06.901223 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:06.916503 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:07.069571 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 14:23:07.161445 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:07.306249 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 14:23:07.317030 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:07.334121 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:07.344091 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:07.353168 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:07.357455 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:07.369057 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:07.387200 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 14:23:07.393760 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:23:07.395068 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:07.553102 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:23:07.571166 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:07.575284 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:23:07.579355 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:07.615859 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 14:23:07.618981 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:07.620294 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:07.623381 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:07.719656 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:07.814482 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:07.816285 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:07.827365 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:07.844274 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:07.849561 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240503 14:23:07.860865 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:07.867089 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:08.041481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:08.054348 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:08.054646 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:08.068466 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:08.080463 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:08.226830 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 14:23:08.269202 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:08.284603 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:08.300050 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:08.307444 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:23:08.321705 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:08.325683 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:08.336700 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:08.337857 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:08.346665 6523 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:08.346844 6523 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:08.347685 6523 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:23:08.351270 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:08.355580 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:08.355720 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:08.356087 6523 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:08.375629 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:08.396741 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:23:08.440186 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:23:08.468279 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:23:08.488113 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:23:08.533430 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:08.541877 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:08.557499 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:08.563511 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:08.576318 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:08.767294 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:08.810009 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:08.898461 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:08.900597 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:23:08.981027 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:08.988873 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:09.000401 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:09.013823 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:09.026017 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:09.065243 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240503 14:23:09.190544 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:09.198685 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:09.221688 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:09.248111 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:09.264865 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:09.326830 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:09.377038 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:23:09.398047 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:09.412959 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:23:09.459846 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:09.480259 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:09.484151 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:09.486586 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:09.500624 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:09.501372 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:23:09.691982 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:09.724372 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:09.727427 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:09.731704 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:09.748962 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:09.757294 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:09.856642 6807 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:09.856830 6807 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:09.857537 6807 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:09.859140 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:09.859337 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:09.859442 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:09.859665 6807 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:09.892707 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:23:09.931578 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:09.934357 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:09.942101 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:09.943439 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:09.953691 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:09.975934 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:23:10.014034 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:10.142972 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:23:10.143886 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240503 14:23:10.169526 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:10.180438 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:10.185992 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:10.195555 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:10.197522 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:10.217159 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:10.252131 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:23:10.472437 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:23:10.526394 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240503 14:23:10.563534 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:10.593161 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:10.616066 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:10.622812 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:10.624066 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:10.631006 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:10.797475 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:10.811678 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:10.820886 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:10.823804 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:10.830363 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:10.837430 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:10.855096 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:10.863260 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 14:23:10.991696 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:23:11.006232 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:11.040773 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:11.046834 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:11.051661 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:11.054072 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:11.107122 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:11.139667 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:23:11.266180 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:11.286732 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:11.298789 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:11.309036 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:11.324373 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:11.350347 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 14:23:11.366896 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:11.391525 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:23:11.480666 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240503 14:23:11.504156 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:11.536350 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:11.548457 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:11.562317 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:11.581182 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:11.595537 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:11.776496 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:11.778080 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:11.798579 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:11.801182 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:11.805931 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:11.829540 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:11.868786 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:23:11.940729 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:23:12.028764 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:23:12.096477 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:12.277575 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:12.280526 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:12.302701 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:12.333236 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:12.339510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:12.340504 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:12.394022 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240503 14:23:12.530660 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:12.544157 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:12.544857 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:12.549191 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:23:12.551441 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:12.575385 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:12.590231 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:23:12.616956 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 14:23:12.691011 6523 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:12.691236 6523 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:12.692112 6523 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:12.692780 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:23:12.693038 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:23:12.693473 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:12.696238 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:12.696365 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:12.696604 6523 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:12.698449 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:12.764499 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:12.776940 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:23:12.791437 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:12.793205 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:12.793187 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:12.813947 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:12.839313 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:23:12.892163 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:12.977623 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 14:23:12.994781 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:13.013976 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:13.014802 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:13.017377 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:13.027936 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:13.081692 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:23:13.158934 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:13.204161 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:13.204769 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:13.206068 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:13.211855 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:13.224412 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:13.294448 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:23:13.449846 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:13.456641 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:23:13.457976 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:23:13.464030 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:13.466671 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 14:23:13.484602 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:13.506971 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:13.510769 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:13.522002 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:13.547279 6807 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:13.547487 6807 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:13.548144 6807 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:13.551299 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:23:13.551664 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:13.551789 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:13.552264 6807 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:13.637823 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:13.802279 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:23:13.923413 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:13.928710 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:13.936543 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:13.947467 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240503 14:23:13.967227 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:13.969427 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:13.996790 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:14.032420 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:23:14.120085 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:23:14.132493 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 14:23:14.168049 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:14.177501 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:14.179637 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:14.184706 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:14.209867 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:14.241966 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:14.393536 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:14.399123 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:14.419301 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:14.435322 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:14.440112 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:14.439546 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:14.507984 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:23:14.644866 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:14.652642 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:14.661069 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:14.661463 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:14.684995 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:14.691740 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:14.813201 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 14:23:14.829356 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:14.874370 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:14.883476 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:14.890275 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:14.908674 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:14.911432 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:15.029670 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:23:15.048746 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:23:15.094763 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:15.097878 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:15.117091 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 14:23:15.117532 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:15.130487 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:15.134243 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:15.159531 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:15.244745 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:23:15.245661 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:15.303071 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:23:15.362174 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240503 14:23:15.531829 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:15.544544 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:15.557096 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:15.569566 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:15.574360 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:15.619401 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:15.671627 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:15.698863 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 14:23:15.713802 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 14:23:15.801512 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:15.810051 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:23:15.810969 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:15.823374 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:15.831490 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:15.836817 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:15.994779 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:23:16.000989 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:23:16.025523 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:16.039167 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:16.057796 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:16.071882 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:16.078174 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:16.084758 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:16.140519 6807 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:16.140727 6807 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:16.141448 6807 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:16.145233 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:23:16.146597 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:16.146754 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:16.147019 6807 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:16.161281 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:16.273010 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:16.290006 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:16.307228 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:16.308813 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:16.337571 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240503 14:23:16.341835 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:16.481999 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:16.509320 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 14:23:16.515429 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:23:16.523520 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:16.543758 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:16.544916 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:16.563189 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:16.563364 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:16.574069 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:16.690305 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:16.794266 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:16.804598 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:16.820048 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:16.844951 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:16.846882 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:16.975751 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:23:17.101591 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:17.220688 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:17.234865 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:17.236641 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:17.250383 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:17.258863 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:17.285277 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:17.427776 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240503 14:23:17.457747 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:17.469290 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:17.485761 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:17.489941 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:17.492561 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:17.541949 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:17.570741 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 14:23:17.577558 7956 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:17.577730 7956 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:17.578500 7956 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:17.579305 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:23:17.579542 2299 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:23:17.580381 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:17.580932 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:17.581030 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:17.581506 7956 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:17.604540 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:17.611163 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:23:17.674230 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:23:17.687934 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:17.692575 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:17.696545 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:17.697252 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:17.702307 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:17.703974 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240503 14:23:17.713593 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:17.736569 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 14:23:17.894945 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:23:17.924980 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:17.935643 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:17.938267 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:17.963344 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:17.986282 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:18.058531 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:23:18.101783 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 14:23:18.140472 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:18.143438 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 14:23:18.146535 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:18.151150 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:18.167377 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:18.184927 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:18.195663 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:18.212648 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:18.368363 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:18.383034 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:18.399149 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:18.401695 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:18.414695 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:18.418092 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:23:18.564826 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 14:23:18.570971 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:23:18.626067 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:18.731454 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:23:18.751716 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:18.825496 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:18.851069 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:18.863916 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:18.884510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:18.888063 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:18.987407 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:23:19.086827 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 14:23:19.102975 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:19.108572 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:19.114688 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:19.119275 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:19.135526 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:19.183072 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240503 14:23:19.220978 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:19.249164 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:19.337965 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:19.344590 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:19.345449 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:19.353754 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:19.361146 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:19.477012 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:23:19.560447 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:19.561937 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:19.587464 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:19.588701 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:19.601436 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:19.662253 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:19.664233 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:19.811125 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:19.811625 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:19.818681 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:19.833009 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:19.851994 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:19.874256 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 14:23:20.030369 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:20.036720 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:20.058281 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:20.071887 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:20.074673 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:23:20.074800 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:20.196216 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:20.208845 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:20.272255 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:23:20.273180 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:23:20.275094 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:23:20.340611 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240503 14:23:20.394250 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240503 14:23:20.445747 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:23:20.489898 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:20.519810 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:20.523712 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:20.546362 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:20.572067 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:20.654523 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240503 14:23:20.656301 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:20.702169 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:20.743896 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:20.753329 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 14:23:20.754926 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:20.770952 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:20.777922 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:20.837498 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:20.895875 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:23:21.025311 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:21.030627 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:21.039994 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:21.040568 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:21.050453 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:21.172324 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:21.220458 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:21.235438 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:21.236713 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:21.248131 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:23:21.253558 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:21.275240 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:21.280424 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:21.318553 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:23:21.450146 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:23:21.504974 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:21.506404 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:21.510849 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:21.514722 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:21.532092 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:21.626937 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240503 14:23:21.640676 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:21.742822 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:21.756228 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:21.757239 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:21.767571 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:21.789045 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:21.799695 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:23:21.846266 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:22.088305 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:23:22.182467 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:22.196019 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:22.199014 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:22.217372 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:22.221011 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:22.229874 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:22.237234 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:22.326104 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 14:23:22.423832 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:22.450498 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:22.450909 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240503 14:23:22.471433 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:22.473455 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:22.489538 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:22.561633 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:22.589517 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:23:22.683060 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:22.684952 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:22.696269 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:22.700350 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:22.708581 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:22.715097 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:23:22.792954 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:22.874074 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240503 14:23:22.907608 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:22.913293 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:23:22.925787 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:22.930895 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:22.944401 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:23:22.952278 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:22.972879 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:23:22.983804 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:23.012657 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 14:23:23.028932 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:23.059988 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:23:23.145632 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:23.161583 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:23.165683 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:23.175433 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:23.178123 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:23.318295 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240503 14:23:23.389133 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:23.410869 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:23.411262 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:23.414604 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:23.419147 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:23.431125 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:23.563143 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 14:23:23.618176 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:23.628126 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 14:23:23.789925 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:23:23.817770 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:23.828787 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:23.833195 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:23.840692 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:23.841111 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:23.867496 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:23:23.911399 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:23.972642 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240503 14:23:24.045457 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:24.056515 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:24.060516 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:24.061672 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:24.065065 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:24.073521 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:24.074671 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:23:24.268537 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:24.278673 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:24.299228 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:24.304019 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:24.338531 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:24.415891 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:24.517712 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:24.533071 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:24.552608 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:24.559386 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:24.559741 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:24.563648 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:24.638978 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:23:24.647322 6523 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:24.647485 6523 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:24.648234 6523 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:24.648576 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:23:24.648845 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:23:24.650223 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:24.650627 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:24.650748 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:24.650977 6523 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:24.679440 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 14:23:24.750977 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:24.773248 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:24.784304 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:24.788467 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:24.789352 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:24.926985 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:24.971012 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:23:24.986398 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:24.992712 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:25.002523 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:25.009820 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:25.023101 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:25.049592 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:25.198509 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:23:25.374830 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240503 14:23:25.401064 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 14:23:25.402714 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:23:25.417623 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:25.435648 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:25.449563 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:25.455034 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:25.465812 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:25.522615 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:25.560649 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:23:25.560827 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:23:25.629173 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:23:25.664258 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:25.683291 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:25.693749 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:25.702133 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:25.708179 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:23:25.721343 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:25.769390 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:25.887166 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:25.910244 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 14:23:25.926231 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:25.944176 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:25.959494 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:25.966565 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:25.984373 6523 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:25.984581 6523 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:25.985546 6523 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:25.989573 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:26.004256 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:23:26.013511 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:26.013722 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:26.014084 6523 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:26.014127 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:26.022419 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:23:26.161286 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:26.169210 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:26.186501 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:26.191407 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:26.201347 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:26.288287 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:26.337210 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:26.404878 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:26.410081 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:26.410481 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:26.412335 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:26.422843 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:26.486151 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:23:26.542770 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240503 14:23:26.558754 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:23:26.573608 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:23:26.631690 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:26.635229 9019 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:26.635414 9019 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:26.636122 9019 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:26.638278 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:26.641021 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:26.641137 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:26.641988 9019 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:26.649034 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:26.675033 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:26.675061 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:26.693518 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:26.713668 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:26.839851 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:27.089366 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:27.100214 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:27.100643 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:27.104619 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:27.115458 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:27.163733 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:23:27.231837 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:27.278641 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 14:23:27.336021 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:27.361755 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:27.378985 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:27.380599 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:27.383387 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:27.383904 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:27.457957 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240503 14:23:27.598017 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:27.619351 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240503 14:23:27.625538 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:27.630092 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:27.630659 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:27.631898 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:27.642340 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 14:23:27.647369 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:27.837280 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:23:27.839076 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:27.855149 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:27.855960 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:27.856621 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240503 14:23:27.864151 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:27.875507 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:27.875977 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:27.895431 2149 maintenance_manager.cc:419] P 13adeda3db494b478178ddd5ee8fdb98: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033355
I20240503 14:23:27.915309 2148 maintenance_manager.cc:643] P 13adeda3db494b478178ddd5ee8fdb98: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.005s sys 0.008s Metrics: {"bytes_written":2079,"delete_count":0,"lbm_write_time_us":206,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":12913,"threads_started":1,"update_count":32,"wal-append.queue_time_us":11556}
I20240503 14:23:27.916173 2149 maintenance_manager.cc:419] P 13adeda3db494b478178ddd5ee8fdb98: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.136785
W20240503 14:23:27.928464 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:23:27.955564 2148 maintenance_manager.cc:643] P 13adeda3db494b478178ddd5ee8fdb98: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.020s sys 0.008s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15413,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":14331,"lbm_read_time_us":336,"lbm_reads_lt_1ms":17,"lbm_write_time_us":349,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":14114,"threads_started":2,"update_count":32}
W20240503 14:23:28.016989 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240503 14:23:28.050068 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:28.071992 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:28.082203 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:28.088860 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:23:28.098487 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:28.099736 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:28.117997 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:28.194921 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:23:28.203794 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:23:28.270681 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 14:23:28.305639 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:28.312039 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:28.313860 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:28.319711 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:28.322017 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:28.335016 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:28.421135 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 14:23:28.562809 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:23:28.644425 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:28.656304 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:23:28.727968 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:28.728796 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:28.729588 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:28.730556 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:28.753366 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:28.781415 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:28.820734 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:23:28.900230 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240503 14:23:28.975478 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:29.003509 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:29.003891 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:29.007838 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:29.009789 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:29.056715 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:23:29.069440 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:29.217051 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:29.219565 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:29.237578 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:29.267936 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:29.270938 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:29.342437 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:29.344961 9384 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:29.345122 9384 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:29.345790 9384 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:29.347764 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:29.358121 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:29.358228 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:29.358476 9384 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:29.475939 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:29.486030 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:29.494092 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:29.503603 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:29.511267 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:29.524276 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:29.688450 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:29.695855 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:29.726542 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 14:23:29.727077 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:29.728302 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:23:29.734730 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:29.751665 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:29.824259 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:29.900220 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:29.962504 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:29.982983 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:29.983018 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:29.983382 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:29.984421 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:30.021714 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 14:23:30.183948 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:23:30.317319 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:30.382954 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:30.416198 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 14:23:30.420657 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:30.422911 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:23:30.423986 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:30.424639 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:30.444689 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:30.465510 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:30.534598 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:23:30.645431 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:23:30.653879 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:30.666532 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:30.677572 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:30.688958 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 14:23:30.700261 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:30.705708 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:30.729769 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:23:30.794780 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:23:30.826920 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 14:23:30.882362 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:30.899551 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:30.900643 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:30.902287 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:30.919732 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:30.926870 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:30.931456 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:30.948330 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 14:23:30.958748 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:23:31.159545 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:31.162097 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:31.176599 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:31.180699 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:31.207280 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:31.323004 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:31.385933 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:31.387459 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:31.410988 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:31.422326 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:31.433475 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:31.519779 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:31.538861 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240503 14:23:31.547792 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:23:31.612834 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:31.626269 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:31.633060 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:31.634757 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:31.649070 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:23:31.672775 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:31.819329 9384 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:31.819525 9384 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:31.820379 9384 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:31.821702 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:31.821919 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:31.822041 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:31.822263 9384 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:31.831789 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:31.982088 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:32.048907 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:32.050629 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:32.066181 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:32.071094 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:32.072028 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:32.102852 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 14:23:32.165223 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:23:32.253813 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:32.278632 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:32.289985 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:32.295645 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:32.311690 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:32.355369 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:32.490165 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:23:32.492281 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 14:23:32.494400 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:32.507386 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:32.524183 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:32.525540 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:32.531730 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:32.553030 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:23:32.565814 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:32.732113 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:32.733038 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:32.753783 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:32.764681 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:32.775825 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:32.898252 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:23:32.905673 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:32.914517 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 14:23:32.975553 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:23:32.978070 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:32.988612 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:32.993270 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:32.996004 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:32.998512 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:33.007673 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:33.157120 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:23:33.185642 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:33.194923 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:33.203495 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 14:23:33.212056 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:33.221123 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:33.234359 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:33.284564 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:23:33.298372 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:23:33.353636 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:33.464241 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 14:23:33.502665 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:33.552964 9986 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:33.553150 9986 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:33.553978 9986 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:23:33.556478 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:33.556588 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:33.556653 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:33.557085 9986 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:33.567093 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:33.579741 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:23:33.628412 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 14:23:33.639765 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:33.647910 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:33.648376 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:33.669610 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:33.688417 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:33.701503 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 45 similar messages]
W20240503 14:23:33.802186 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 14:23:33.803278 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5f631c7713d24f4caf7550bb05147615: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:33.831470 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:33.884048 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:23:33.887840 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:33.889052 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:33.897754 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:33.898689 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:33.901844 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:33.908921 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:33.918211 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240503 14:23:34.113234 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:34.114056 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:34.118731 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:34.121127 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:34.141314 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:34.147809 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:23:34.221033 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:34.338585 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:34.388911 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:34.408705 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:34.412072 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:34.412953 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:34.451047 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:34.576566 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 14:23:34.607123 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:34.619372 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:34.624235 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:34.626320 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:34.641247 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:23:34.647927 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:34.649986 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:34.730980 9986 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:34.731228 9986 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:34.732165 9986 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:34.732569 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:23:34.732793 2692 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:23:34.733785 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:34.740315 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:34.740437 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:34.740667 9986 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:34.850313 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:34.851724 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:34.852775 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:34.860635 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:34.864149 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:34.958714 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:35.000110 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 14:23:35.074563 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:35.107406 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240503 14:23:35.276912 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:35.287536 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:35.289017 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 14:23:35.306074 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:35.317155 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:35.332981 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:23:35.341729 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:35.412487 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:35.419831 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:23:35.431747 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 14:23:35.460235 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:23:35.517154 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:35.527073 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:35.538638 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:35.548965 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:35.549582 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:35.565697 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:35.583472 10191 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:35.583649 10191 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:35.584527 10191 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:35.591279 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:35.592015 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:35.592116 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:35.592552 10191 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:35.624055 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240503 14:23:35.761176 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:35.771337 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:35.779259 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:35.780138 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 14:23:35.781991 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:35.811205 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:35.827198 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 14:23:35.892735 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 14:23:35.923255 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:35.987486 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:36.008960 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:36.021003 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:36.030290 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:36.031338 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:36.038692 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:36.184617 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:23:36.225636 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:36.233228 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:36.260583 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:36.261297 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:36.264051 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:36.312515 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:23:36.363325 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240503 14:23:36.374442 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:36.462797 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:36.480973 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:36.492337 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:36.496479 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:36.509306 10743 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:36.509477 10743 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:36.510211 10743 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:23:36.510572 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:36.516157 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:36.517926 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:36.549530 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:36.549679 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:36.549952 10743 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:36.740140 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 14:23:36.838235 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:36.884004 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:36.947590 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:36.959108 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:37.013404 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:37.028605 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:37.040314 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:37.195894 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:23:37.224040 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:37.239086 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240503 14:23:37.240695 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:37.250993 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:23:37.266274 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:37.272840 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:37.283154 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:37.399317 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:37.477970 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:37.479442 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 14:23:37.494670 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:37.497186 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:37.501610 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:37.507120 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:37.507928 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:37.587437 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240503 14:23:37.588379 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 14:23:37.727171 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:37.728435 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:37.733446 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:37.743084 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:37.744693 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:23:37.769695 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 14:23:37.774933 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:37.886812 9986 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:37.887030 9986 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:37.888012 9986 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:37.888252 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:23:37.888473 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:23:37.889881 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:37.890018 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:37.890156 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:37.890444 9986 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:37.909453 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 14:23:37.934517 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:37.953946 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:37.956517 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:37.986459 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:37.987229 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:37.997792 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:38.055466 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 14:23:38.071144 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:23:38.090525 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:38.107434 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:23:38.181183 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:38.188463 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:38.195920 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:38.208710 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:38.214598 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:38.284464 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 14:23:38.358304 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240503 14:23:38.502413 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:38.547369 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:38.631222 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:38.639441 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:38.643772 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:38.650254 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:38.656168 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:38.746297 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240503 14:23:38.770193 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:23:38.809280 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240503 14:23:38.851617 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:38.870182 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:38.871486 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:38.892880 10957 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:38.893081 10957 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:38.893795 10957 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:23:38.895299 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:38.915625 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:38.920097 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:38.924075 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:38.924341 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:38.924718 10957 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:38.951965 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:23:39.042104 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:39.055461 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:39.116552 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:39.125875 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:39.128424 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:39.135043 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:39.136893 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:39.220036 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 14:23:39.319717 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:39.331408 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:39.333171 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:39.335800 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:39.344251 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:39.424412 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:39.517199 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:39.531292 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240503 14:23:39.545691 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:39.579376 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:39.579670 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:39.589134 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:39.591463 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:39.786518 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:39.791833 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:39.796368 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:39.802490 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:39.813647 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:39.849517 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:23:39.951483 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:23:39.962595 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 14:23:40.042246 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:40.082566 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:40.098824 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240503 14:23:40.210355 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:40.214917 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:40.232816 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:40.235504 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:40.238886 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:40.298213 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:23:40.450162 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:40.463541 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:40.466609 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:40.470885 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:40.478077 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:40.505321 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 14:23:40.510931 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:40.531677 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 14:23:40.600641 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:23:40.601511 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:40.627462 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:23:40.661077 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:23:40.680873 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:40.696122 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:40.702626 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:40.703584 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:40.705394 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:40.764797 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 14:23:40.849038 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:40.918761 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:40.925724 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:40.930516 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:40.952086 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:40.960664 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:41.025504 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 14:23:41.087306 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:41.171293 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:41.187484 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:41.188913 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:41.206195 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:41.213043 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:41.260960 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:41.308315 2146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 14:23:41.309494 2146 catalog_manager.cc:4573] TS 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 470b425f5cef494ab3334fe2b85e1c4e: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:41.310241 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:41.396622 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240503 14:23:41.405157 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:41.432257 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:41.442200 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:23:41.451860 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:41.453474 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:41.486598 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:41.533039 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:41.592905 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:23:41.811024 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:41.821441 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 14:23:41.870481 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:41.876636 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:41.900226 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:41.900867 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:41.917935 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:41.927841 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:42.098250 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:42.101135 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:42.119866 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:42.120035 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:42.126626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:42.296792 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:42.348345 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:42.366871 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:42.373580 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:42.377422 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:42.380944 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:42.403507 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:42.499351 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:42.530990 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 14:23:42.537621 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240503 14:23:42.609097 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:42.619463 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:42.621521 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240503 14:23:42.623776 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:42.624366 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 14:23:42.627671 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:42.641346 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:42.780612 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:42.819485 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:42.831131 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:42.846271 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:42.851622 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:42.853664 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:42.903935 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:42.965260 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:42.981832 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:23:42.982662 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:43.030992 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 14:23:43.034135 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240503 14:23:43.053567 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:43.063566 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:43.079352 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:43.093204 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:43.099963 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:43.127619 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:43.356559 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:43.370832 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:43.491683 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:43.494473 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:43.499439 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:43.519649 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240503 14:23:43.520404 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:43.528010 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:43.705202 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:23:43.712108 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:43.721122 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:43.732363 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:43.737298 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:43.744709 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:43.788594 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 14:23:43.809649 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:43.858310 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 14:23:43.860452 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240503 14:23:43.900868 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:43.921384 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:43.937163 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:43.937530 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:43.966089 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:43.974682 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:44.006247 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:23:44.052827 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 14:23:44.065874 2147 catalog_manager.cc:4573] TS c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet abdff0b719a342cfbde746751bd2a89d: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:44.179471 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:44.180768 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:44.196102 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:44.197978 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:44.212356 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:44.381714 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:44.388527 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:44.399545 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:44.403254 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:44.417080 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:44.419062 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:44.422652 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 14:23:44.438338 9986 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:44.438535 9986 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:44.439829 9986 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:23:44.447676 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:44.465624 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:44.465870 2234 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:44.466202 9986 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:44.475009 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:44.480095 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:44.631834 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:44.638660 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:44.649562 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:44.653153 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:44.696906 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:44.735952 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240503 14:23:44.829914 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:23:44.872251 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:44.960912 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:45.073200 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:45.115496 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:45.116313 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:45.121407 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:23:45.122331 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:45.130841 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:45.321270 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:45.323462 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:45.329712 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:23:45.335448 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:45.336951 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:45.342176 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:45.386225 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 14:23:45.390684 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:45.446105 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 14:23:45.455441 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:45.500413 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:23:45.507565 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 14:23:45.566428 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:45.587720 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:45.599462 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:45.610505 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:45.612192 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:45.626401 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:23:45.676162 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:23:45.806660 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:45.807590 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:45.814932 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:45.823968 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:45.827837 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:45.838531 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:45.978919 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:46.020735 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:23:46.021672 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:46.042096 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:46.042582 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:46.051275 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:46.059381 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:46.106724 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 14:23:46.246634 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:46.251158 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:46.252034 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:46.253039 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:46.277375 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:46.296641 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 14:23:46.435003 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:46.444653 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240503 14:23:46.447247 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:46.540465 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:46.621049 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:23:46.655318 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:46.667385 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:46.677516 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:46.694783 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:46.728709 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:46.936185 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:46.944200 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:46.955792 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:46.962720 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:46.962721 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:46.976130 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:46.994896 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:47.098363 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 14:23:47.186029 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:47.203766 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:47.211979 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:47.212991 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:47.213145 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:47.305733 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 14:23:47.426522 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:47.444413 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:47.458712 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:47.481421 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:47.483845 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:23:47.491428 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:47.506743 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:47.507066 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:47.540138 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 14:23:47.644390 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:47.661253 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:47.681802 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:47.695115 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:47.696170 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:47.841295 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240503 14:23:47.842157 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 14:23:47.845925 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:23:47.906716 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 14:23:47.933339 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:47.933933 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:47.954412 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:47.954990 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:23:47.955848 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:47.975775 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 14:23:47.977993 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:48.028156 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:23:48.066291 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:48.118525 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:48.168167 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240503 14:23:48.377715 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:48.393189 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:48.393801 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:48.399358 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:48.422935 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:48.454177 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:48.472417 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 14:23:48.575469 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 14:23:48.591225 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:48.595077 9986 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:48.595361 9986 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:48.596482 9986 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:23:48.598439 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:48.605016 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:48.622606 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:48.623198 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:48.629595 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:48.629820 2237 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: c2cf5d2244244a2db5ebc7dc46fd430d, d4cbc83202c8473d8c897582d30e5989
W20240503 14:23:48.630033 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:48.630807 9986 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:48.631270 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:48.642187 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:48.668848 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:48.694188 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240503 14:23:48.821458 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:48.834611 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:48.840607 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:48.845300 12050 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:48.845544 12050 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:48.846444 12050 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:48.852541 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:48.856462 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:48.869315 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:48.899669 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:48.901834 2498 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:48.902125 12050 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:48.902854 2498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:23:49.046761 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:49.052927 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:49.054937 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:49.059518 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:49.065061 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:49.075820 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:49.078547 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:49.149298 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:23:49.242190 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:49.258509 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:49.258857 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:49.260903 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:49.263698 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:49.386204 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:23:49.492715 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:49.508348 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:49.511930 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:49.513635 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:49.518671 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:49.625905 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:49.674427 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:49.756134 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 14:23:49.953243 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:49.955157 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:49.974138 12050 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:49.974329 12050 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:49.975026 12050 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:49.977056 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:49.979061 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:49.990798 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:49.994638 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:49.994936 12050 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:50.000794 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:50.001067 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:23:50.005937 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:50.051096 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 14:23:50.062750 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:50.076493 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:50.114368 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 14:23:50.173565 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:50.190161 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:50.219416 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:50.226311 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:50.227707 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 14:23:50.253659 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:50.321259 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:50.398619 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:50.398919 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 14:23:50.457784 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:50.479487 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:50.484283 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:50.485670 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:50.505267 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:50.550997 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:50.609234 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 14:23:50.680375 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:50.684377 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:50.703032 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:50.703702 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:50.706388 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:50.713888 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:50.796831 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:50.808565 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240503 14:23:50.866014 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 14:23:50.898720 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:50.910786 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:50.924521 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:50.929803 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:50.940374 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:50.956490 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:51.107278 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 14:23:51.125494 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:51.137846 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:51.141566 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:23:51.145090 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:51.148509 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:51.149047 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:51.199221 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:51.321520 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:51.413803 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240503 14:23:51.537658 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:51.551548 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:51.557700 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:51.563381 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:51.613749 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:51.689702 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:23:51.769649 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:51.779955 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:51.785820 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:51.790688 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:51.804374 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:51.804829 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:51.915737 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:51.991428 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 14:23:52.006968 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:52.016497 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:52.017324 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:52.019691 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:52.048424 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:52.190884 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:52.194273 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 14:23:52.227890 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:52.228480 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:52.253543 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:52.258826 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:52.273741 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:52.291834 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 14:23:52.365578 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:52.398708 2144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240503 14:23:52.403661 2144 catalog_manager.cc:4573] TS d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet eb1d30b76c4d43d0a54a49892d09a6f8: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:23:52.411607 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:52.478053 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:52.484349 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:52.498514 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:52.505867 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:52.507735 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:52.567552 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 14:23:52.614804 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:52.632081 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:23:52.674870 12720 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:23:52.675060 12720 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:23:52.675935 12720 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:23:52.678395 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:52.679693 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:23:52.679805 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:23:52.680171 12720 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:23:52.701390 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:52.720808 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:52.726378 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:52.740314 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:52.757164 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:52.844195 2365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:23:52.845127 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 14:23:52.856586 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:23:52.969391 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:53.057668 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:53.144479 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:53.166271 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:53.173197 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:53.176414 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:53.179826 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:53.199524 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 14:23:53.314352 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:23:53.324139 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 33 similar messages]
I20240503 14:23:53.349653 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:53.359714 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:53.361315 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:53.383904 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:53.384969 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:53.492959 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:23:53.507315 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:53.508414 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:23:53.534749 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 14:23:53.580456 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:53.584645 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:53.603541 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:53.610234 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:53.617856 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:53.639873 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:53.662467 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 14:23:53.836587 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:53.840405 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:53.845901 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:53.848992 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:53.874919 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:53.955399 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:53.959867 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240503 14:23:53.963632 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:54.045365 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:54.047082 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240503 14:23:54.062306 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:54.064990 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:54.067428 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:54.103127 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:54.172156 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:54.280051 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:54.309478 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:54.309907 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:54.315685 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:54.344260 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:54.420718 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:54.481851 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:23:54.653736 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:54.727028 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
I20240503 14:23:54.729346 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:54.753583 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:54.755718 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:54.798394 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:54.799010 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 14:23:54.811957 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:54.847081 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 14:23:54.952311 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:54.987771 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:54.992648 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 14:23:55.005916 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:55.015853 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:55.019448 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:55.035108 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:55.093561 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:23:55.094559 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:55.119554 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:23:55.207739 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:55.219339 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:55.231953 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:55.235347 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:55.253067 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:55.328619 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 14:23:55.476321 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:55.478042 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:55.481427 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:55.499485 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:55.521127 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:55.521733 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:55.583811 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:55.601902 13417 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:23:55.602063 13417 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:23:55.603119 13417 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:23:55.604377 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:23:55.604661 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
W20240503 14:23:55.605968 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:55.608273 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:55.608402 2237 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:23:55.608695 13417 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:23:55.685391 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 14:23:55.713577 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:55.720542 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:55.732105 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:55.738379 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:23:55.739753 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:55.742909 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:55.864583 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:55.868705 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:23:55.936812 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:55.956423 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:23:55.962661 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:55.966635 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:55.972275 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:55.973062 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 14:23:56.003839 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:56.016955 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:23:56.067090 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:56.272928 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 14:23:56.374516 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:23:56.384577 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:23:56.390836 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:56.402588 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:56.408257 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:56.411943 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:56.413621 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:56.512907 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:56.631559 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:56.649909 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:56.659368 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:56.670338 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:56.691591 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:56.706964 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240503 14:23:56.827016 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:56.915036 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:56.930186 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:56.939064 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:56.954525 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:56.960177 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:57.012380 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:57.175513 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:23:57.176506 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:57.177112 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:57.177173 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:57.181252 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:57.186189 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:57.294361 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:57.385571 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:57.393877 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:57.397951 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:57.407703 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:57.413699 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240503 14:23:57.429610 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:57.456629 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:57.493726 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 14:23:57.528793 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 14:23:57.618467 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:57.620308 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:23:57.629309 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:57.644120 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:57.649993 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:57.662055 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 14:23:57.662799 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:57.684520 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 14:23:57.690631 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:57.711393 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:23:57.941725 2367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:23:58.011631 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:58.050740 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:58.050792 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:58.071319 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:58.087673 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:58.088064 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:58.143715 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 14:23:58.257623 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:23:58.258983 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:58.268383 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:58.280917 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:58.287039 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:58.299788 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:58.425828 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:58.487087 2237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:23:58.488519 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 14:23:58.495746 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:58.506983 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:58.507494 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:58.525002 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:23:58.532099 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:23:58.532488 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:23:58.536944 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:58.645337 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:58.680769 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:23:58.715421 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:58.722518 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:58.746958 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:58.752249 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240503 14:23:58.758729 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:58.765309 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:58.964622 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:58.980826 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:58.981825 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:58.983024 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:23:58.993219 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:59.000641 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240503 14:23:59.033190 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 14:23:59.040696 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:23:59.052109 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:59.160300 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 14:23:59.222808 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:59.239730 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:59.240883 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:59.248989 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:59.254667 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:59.455371 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:23:59.554653 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:23:59.651280 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:59.667135 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:59.669625 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:59.688099 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:23:59.705456 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:59.776892 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 14:23:59.785822 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:59.797677 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 14:23:59.880210 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:23:59.882032 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:23:59.883584 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:23:59.887212 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:23:59.898320 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:23:59.905894 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:23:59.942468 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:23:59.942907 2366 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.1:33597, user_credentials={real_user=slave}} blocked reactor thread for 46723.5us
W20240503 14:23:59.966356 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:00.116616 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:00.120488 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:00.121400 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:00.129590 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:00.130790 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:00.136507 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 14:24:00.206735 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 14:24:00.234030 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:24:00.332190 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:00.332732 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:00.333134 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:00.334681 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:00.339504 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:00.364039 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:24:00.419203 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:00.491669 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:00.534960 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:00.543711 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:00.578884 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:00.582609 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 14:24:00.585346 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:00.589653 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:00.756134 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:00.791999 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:00.796460 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:00.796794 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:00.810784 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:00.826200 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 14:24:00.841457 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:00.905242 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:24:00.983527 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:24:01.074602 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:01.198715 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:01.217985 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:01.223332 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:24:01.224488 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 14:24:01.225560 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:01.226511 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:01.238459 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:01.333662 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:01.373389 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240503 14:24:01.405441 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:24:01.447347 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:01.452179 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:01.469312 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:01.479774 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:01.483007 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:01.511853 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:01.514467 13147 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:24:01.514631 13147 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:24:01.522126 13147 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:24:01.524720 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:01.532081 13417 raft_consensus.cc:482] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:24:01.532251 13417 raft_consensus.cc:504] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:24:01.533354 13417 leader_election.cc:288] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
W20240503 14:24:01.541615 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
I20240503 14:24:01.541819 2495 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:24:01.542162 13147 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:24:01.548242 2234 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:24:01.548561 2236 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:01.548656 2236 leader_election.cc:302] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:24:01.548918 13417 raft_consensus.cc:2726] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:24:01.559535 2237 leader_election.cc:334] T 78edc267e9f44d82b7dbb663628359fe P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:01.611657 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:24:01.723878 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:01.755846 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:01.778923 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:01.779752 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:01.816802 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:01.825112 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:01.948187 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:02.006480 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:02.015976 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:02.022349 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:02.031580 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:02.036953 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:02.134863 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 14:24:02.157701 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:24:02.197602 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:02.215790 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:02.217505 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 14:24:02.223666 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:02.239507 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:02.242626 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:02.323822 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240503 14:24:02.399878 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:02.420063 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:02.438656 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:02.453318 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:02.462020 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:02.477118 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:02.500190 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:02.596808 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 14:24:02.625020 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 14:24:02.672080 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:24:02.716024 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:24:02.791316 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:02.891140 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:02.897542 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:02.899318 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:02.908447 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:02.917856 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:03.096370 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:03.100649 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240503 14:24:03.102970 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:03.107350 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:03.109095 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:03.114859 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:03.118834 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:03.121034 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 14:24:03.166275 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 14:24:03.180646 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:03.329861 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:03.351228 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:03.352731 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:03.354519 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:03.391489 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:03.468757 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:24:03.478260 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:03.504913 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:24:03.562657 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:03.573088 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:03.573815 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:03.596025 14538 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:24:03.596213 14538 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:24:03.596961 14538 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
I20240503 14:24:03.597676 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:03.599622 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:24:03.599900 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:03.599995 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:24:03.600543 14538 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:24:03.626048 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:03.647984 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:03.667445 2234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 33 similar messages]
W20240503 14:24:03.718333 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 14:24:03.763368 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240503 14:24:03.830631 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:03.831822 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:03.831981 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:03.837211 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:03.862263 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:04.023332 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:04.030591 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:04.031834 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:24:04.042918 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:04.060464 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:04.061061 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:04.079445 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:04.087651 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 14:24:04.156953 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:04.460094 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:04.469810 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:04.480697 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:04.484519 14777 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:24:04.484683 14777 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:24:04.485530 14777 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c37183c2965f46a28847c1262eb97829 (127.20.99.4:39091), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:24:04.486891 2691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "c37183c2965f46a28847c1262eb97829" is_pre_election: true
I20240503 14:24:04.487099 2691 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f22b4d5b1b5466dbbdd344f0721c05f in term 2.
I20240503 14:24:04.488866 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:04.489555 2237 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:04.498067 2236 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:04.498261 2236 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:24:04.498553 14777 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:24:04.501341 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:04.509853 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 14:24:04.544214 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:04.630383 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240503 14:24:04.665393 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:04.680038 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:04.696764 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:04.701490 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:04.709609 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:24:04.712576 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:04.721935 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:04.746711 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 14:24:04.895560 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:04.903993 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:04.911375 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:04.922827 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:04.968034 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:05.039738 14776 raft_consensus.cc:482] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:24:05.039939 14776 raft_consensus.cc:504] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:24:05.040637 14776 leader_election.cc:288] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:24:05.042078 2498 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:05.042763 2497 leader_election.cc:334] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:05.042876 2497 leader_election.cc:302] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:24:05.043339 14776 raft_consensus.cc:2726] T 1d584b7026b64ba1b720ed21d7019740 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:24:05.048470 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:24:05.122072 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:05.128455 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:05.147082 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:05.161269 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:05.161619 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:05.163322 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:05.180333 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:05.196091 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:24:05.250283 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:24:05.320080 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240503 14:24:05.345739 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:05.356118 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:05.357640 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:05.369916 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:05.374471 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:05.437463 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 14:24:05.561625 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:05.567492 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:05.567777 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:05.613123 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:05.613399 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:05.641829 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:05.643669 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:05.765306 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 14:24:05.830200 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240503 14:24:06.001180 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:06.029013 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:06.032897 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:06.036449 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:06.051420 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:06.052076 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:06.090404 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240503 14:24:06.182085 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:24:06.238044 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:24:06.239998 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:06.254736 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:24:06.269168 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:06.274142 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:06.275313 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:06.281991 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:06.282461 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:06.353797 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:24:06.355674 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
I20240503 14:24:06.513813 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:06.522049 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:06.530277 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:06.530287 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:06.542622 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:06.559494 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:06.604521 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:24:06.765122 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:06.768265 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:06.769215 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:06.790319 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:06.794909 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:06.808022 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:06.934684 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:07.000048 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:07.016036 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:07.021102 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:07.028779 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:07.029031 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:07.060956 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 14:24:07.181679 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 14:24:07.188498 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:07.224895 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:07.239606 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:07.240366 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:07.251055 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:07.274996 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:07.276400 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 14:24:07.539705 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:07.651770 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:07.663764 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:07.671274 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:24:07.674574 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:07.675761 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:07.682823 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:24:07.697577 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:07.744199 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:24:07.773373 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:07.812353 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 14:24:07.873670 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:07.875795 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:07.895417 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:07.909544 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:07.910888 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:07.939294 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:08.005312 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 14:24:08.108847 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:08.110590 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:08.129344 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240503 14:24:08.130623 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240503 14:24:08.135391 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:08.142942 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:08.167356 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:08.310874 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:08.347349 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:08.359560 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:08.370764 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:08.394009 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:08.417860 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:08.475095 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 14:24:08.510855 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 14:24:08.525782 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:08.570979 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240503 14:24:08.605012 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:24:08.627076 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:08.633729 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:08.643731 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:08.656791 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:08.669834 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:08.723965 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:24:08.755754 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:08.858919 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:08.863636 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:08.864616 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:08.878957 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:08.880031 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:08.880203 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 14:24:08.891173 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240503 14:24:09.041137 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:09.129179 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 14:24:09.276226 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:09.281512 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:09.310391 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:09.330336 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:09.352134 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:09.359753 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:09.483384 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:09.502892 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240503 14:24:09.531910 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:09.536926 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 14:24:09.556419 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:09.568066 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:09.570855 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:09.584170 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:09.609795 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 14:24:09.738902 15493 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:24:09.739121 15493 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:24:09.740197 15493 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:24:09.740742 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:24:09.740983 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:24:09.742795 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:09.747217 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:09.747370 2627 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:24:09.747817 15493 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:24:09.772436 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:09.791772 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:09.804306 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:09.811264 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:09.824079 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:09.838588 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:09.873134 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:24:10.019412 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:10.023298 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:10.028724 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:10.032733 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:10.033046 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:10.095503 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:10.185806 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:24:10.193029 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:24:10.251506 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:24:10.251683 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:10.253835 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240503 14:24:10.261171 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:10.264842 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:10.271466 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:10.287097 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:10.365401 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:10.495326 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:10.503293 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:10.504086 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:10.516953 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:10.529227 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:10.633229 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240503 14:24:10.698431 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:24:10.719784 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:10.757032 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:10.808912 15725 raft_consensus.cc:482] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4cbc83202c8473d8c897582d30e5989)
I20240503 14:24:10.809128 15725 raft_consensus.cc:504] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b35c386497d046de9be5007a31ab7d10" member_type: VOTER last_known_addr { host: "127.20.99.3" port: 45269 } } peers { permanent_uuid: "d4cbc83202c8473d8c897582d30e5989" member_type: VOTER last_known_addr { host: "127.20.99.7" port: 44387 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } }
I20240503 14:24:10.809875 15725 leader_election.cc:288] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737)
W20240503 14:24:10.811376 2495 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111)
W20240503 14:24:10.811533 2497 leader_election.cc:334] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:10.811681 2497 leader_election.cc:302] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b35c386497d046de9be5007a31ab7d10; no voters: 7146de39e348442da4a363e95fbe0789, d4cbc83202c8473d8c897582d30e5989
I20240503 14:24:10.811929 15725 raft_consensus.cc:2726] T dd294a04366145378e3f5e3d306152e3 P b35c386497d046de9be5007a31ab7d10 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:24:10.864540 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 14:24:10.947800 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:10.947974 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:10.954720 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:10.959475 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:10.960094 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 14:24:10.992033 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:11.025067 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240503 14:24:11.156419 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:11.156500 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:11.163681 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:11.168988 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:11.178337 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 14:24:11.182053 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:24:11.195863 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:11.240103 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:11.266629 2628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 14:24:11.287540 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240503 14:24:11.320619 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:11.367851 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:11.382946 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:11.396416 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:11.402276 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:11.430768 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:11.619385 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:11.625224 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:11.636214 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:11.641444 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:11.642845 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:11.797855 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:11.828161 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 14:24:11.833570 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:11.856875 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:11.858737 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:11.858776 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:11.867352 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:11.867879 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:11.898993 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240503 14:24:12.042912 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:12.058115 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:12.060906 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:12.063663 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:12.065862 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:12.242797 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240503 14:24:12.282341 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 14:24:12.325773 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:12.386379 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:12.463780 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:12.481297 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:12.481393 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:12.501268 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:12.520763 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:12.553573 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:24:12.591007 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:24:12.669610 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:12.680197 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:12.681208 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:12.694931 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:12.702029 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:12.819909 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:12.903386 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:12.916692 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:12.917310 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 14:24:12.918696 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:12.931475 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:12.934849 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:12.938266 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:12.995416 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:24:13.139348 2368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240503 14:24:13.140262 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 14:24:13.146085 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:13.150600 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:13.166061 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:13.178046 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:13.194114 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:13.235625 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 14:24:13.273864 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 14:24:13.330539 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:13.343261 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 14:24:13.343927 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 14:24:13.350085 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:13.363404 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:13.369751 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:13.370110 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:13.375170 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:13.427505 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:13.539400 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:24:13.580000 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:13.589159 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:13.606539 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:13.611548 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:13.627048 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:13.650327 2365 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.99.4:39091, user_credentials={real_user=slave}} blocked reactor thread for 48558.7us
W20240503 14:24:13.731094 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240503 14:24:13.899320 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240503 14:24:13.923135 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:13.931944 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:13.981952 2236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 14:24:14.049932 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:14.050495 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:14.050583 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:14.050635 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:14.068011 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:14.229378 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:14.236812 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:14.255282 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 14:24:14.260748 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:14.262027 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:14.274570 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:14.380914 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:14.429338 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:14.434394 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240503 14:24:14.493752 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:14.505486 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:14.527173 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:14.535108 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:14.571244 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:14.642007 2495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 14:24:14.642856 2495 consensus_peers.cc:485] T abdff0b719a342cfbde746751bd2a89d P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:24:14.742822 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:14.760965 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:14.790745 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:14.791445 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:14.794396 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 14:24:14.857453 2234 consensus_peers.cc:485] T af99824d1853471b867e4ff1f01e9367 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:24:14.862993 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:14.880671 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:14.958701 2365 consensus_peers.cc:485] T 470b425f5cef494ab3334fe2b85e1c4e P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:24:14.984802 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:14.997186 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:15.005896 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:15.008795 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:15.009141 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:15.122757 2365 consensus_peers.cc:485] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240503 14:24:15.221035 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:15.227841 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:15.229727 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:15.230455 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:15.234318 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:15.292176 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:15.363806 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:15.574182 2237 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240503 14:24:15.597681 2236 consensus_peers.cc:485] T 944d2a983d6247498b02a6e7b7daf746 P 7f22b4d5b1b5466dbbdd344f0721c05f -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn't send request to peer 7146de39e348442da4a363e95fbe0789. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240503 14:24:15.634991 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:15.657223 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:15.659236 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:15.676429 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:15.682061 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:15.726940 2498 consensus_peers.cc:485] T 6325a400641b4d25b95592abb4af7d2d P b35c386497d046de9be5007a31ab7d10 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:24:15.734030 2495 consensus_peers.cc:485] T aba6537cdb87400cb8b108b1fbfa6c0f P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:24:15.739426 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:15.760017 2495 consensus_peers.cc:485] T 57c4a133b5c646f78b07f37dc85904f6 P b35c386497d046de9be5007a31ab7d10 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 14:24:15.762565 2368 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240503 14:24:15.782402 2368 consensus_peers.cc:485] T 5f631c7713d24f4caf7550bb05147615 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 14:24:15.819084 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:15.853816 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:15.862341 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:15.868417 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:15.871820 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:15.888296 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:15.919543 2365 consensus_peers.cc:485] T dc88c5dc94844f1c8400c31ad1165a32 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 14:24:16.079026 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:16.101001 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:16.105551 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:16.120514 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 14:24:16.136418 2365 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer d4cbc83202c8473d8c897582d30e5989 (127.20.99.7:44387): Couldn't send request to peer d4cbc83202c8473d8c897582d30e5989. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.7:44387: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240503 14:24:16.136883 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:16.296711 2627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 14:24:16.298712 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:16.299408 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 14:24:16.341719 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
W20240503 14:24:16.347446 2628 consensus_peers.cc:485] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240503 14:24:16.348529 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:16.349485 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:16.354719 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:16.356884 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:24:16.544620 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:16.569661 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 14:24:16.570570 2628 consensus_peers.cc:485] T 832f07a6b12740d08fa67d653b3818bf P c37183c2965f46a28847c1262eb97829 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Couldn't send request to peer c2cf5d2244244a2db5ebc7dc46fd430d. Status: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 14:24:16.582662 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:16.583736 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:16.604501 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:16.731114 2368 consensus_peers.cc:462] T 75bafc7675ca45639b696db6e4b7ce36 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:16.777288 4315 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:24:16.777696 2410 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:24:16.783408 2280 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:24:16.806546 2542 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:24:16.806937 2671 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 14:24:16.883760 2627 consensus_peers.cc:462] T 9974d80e8d31417788e10140de97e722 P c37183c2965f46a28847c1262eb97829 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
I20240503 14:24:16.888125 15768 raft_consensus.cc:482] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c2cf5d2244244a2db5ebc7dc46fd430d)
I20240503 14:24:16.888274 15768 raft_consensus.cc:504] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c37183c2965f46a28847c1262eb97829" member_type: VOTER last_known_addr { host: "127.20.99.4" port: 39091 } } peers { permanent_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" member_type: VOTER last_known_addr { host: "127.20.99.1" port: 33597 } } peers { permanent_uuid: "7146de39e348442da4a363e95fbe0789" member_type: VOTER last_known_addr { host: "127.20.99.5" port: 41737 } } peers { permanent_uuid: "c2cf5d2244244a2db5ebc7dc46fd430d" member_type: VOTER last_known_addr { host: "127.20.99.8" port: 45929 } }
I20240503 14:24:16.889024 15768 leader_election.cc:288] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f22b4d5b1b5466dbbdd344f0721c05f (127.20.99.1:33597), 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737), c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929)
I20240503 14:24:16.889336 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "059a4a35aa94412994bb431342ba0753" candidate_uuid: "c37183c2965f46a28847c1262eb97829" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "7f22b4d5b1b5466dbbdd344f0721c05f" is_pre_election: true
I20240503 14:24:16.889549 2300 raft_consensus.cc:2454] T 059a4a35aa94412994bb431342ba0753 P 7f22b4d5b1b5466dbbdd344f0721c05f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37183c2965f46a28847c1262eb97829 in term 2.
W20240503 14:24:16.890408 2627 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Network error: Client connection negotiation failed: client connection to 127.20.99.5:41737: connect: Connection refused (error 111)
W20240503 14:24:16.890542 2628 leader_election.cc:334] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c2cf5d2244244a2db5ebc7dc46fd430d (127.20.99.8:45929): Network error: Client connection negotiation failed: client connection to 127.20.99.8:45929: connect: Connection refused (error 111)
I20240503 14:24:16.890647 2628 leader_election.cc:302] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 7f22b4d5b1b5466dbbdd344f0721c05f, c37183c2965f46a28847c1262eb97829; no voters: 7146de39e348442da4a363e95fbe0789, c2cf5d2244244a2db5ebc7dc46fd430d
I20240503 14:24:16.890854 15768 raft_consensus.cc:2726] T 059a4a35aa94412994bb431342ba0753 P c37183c2965f46a28847c1262eb97829 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:24:17.128545 2367 consensus_peers.cc:485] T 65a196c723884d068b3728e876c87eb2 P 685c6181561b4056abdecfd6d8bdb3b2 -> Peer 7146de39e348442da4a363e95fbe0789 (127.20.99.5:41737): Couldn'